2023-04-03 05:53:28,883 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b 2023-04-03 05:53:28,897 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-03 05:53:28,916 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-03 05:53:28,916 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/cluster_8445de51-c43b-f77a-d5f2-488a4722dd0f, deleteOnExit=true 2023-04-03 05:53:28,916 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-03 05:53:28,917 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/test.cache.data in system properties and HBase conf 2023-04-03 05:53:28,917 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/hadoop.tmp.dir in system properties and HBase conf 2023-04-03 05:53:28,918 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/hadoop.log.dir in system properties and HBase conf 2023-04-03 05:53:28,918 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-03 05:53:28,919 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-03 05:53:28,919 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-03 05:53:29,041 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-03 05:53:29,449 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-03 05:53:29,454 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-03 05:53:29,454 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-03 05:53:29,454 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-03 05:53:29,454 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-03 05:53:29,455 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-03 05:53:29,455 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-03 05:53:29,456 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-03 05:53:29,456 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-03 05:53:29,456 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-03 05:53:29,457 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/nfs.dump.dir in system properties and HBase conf 2023-04-03 05:53:29,457 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/java.io.tmpdir in system properties and HBase conf 2023-04-03 05:53:29,457 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-03 05:53:29,458 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-03 05:53:29,458 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-03 05:53:30,035 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-03 05:53:30,039 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-03 05:53:30,354 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-03 05:53:30,551 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-03 05:53:30,572 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-03 05:53:30,614 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-03 05:53:30,649 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/java.io.tmpdir/Jetty_localhost_41439_hdfs____46xoyv/webapp 2023-04-03 05:53:30,784 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41439 2023-04-03 05:53:30,826 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-03 05:53:30,826 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-03 05:53:31,250 WARN [Listener at localhost/37989] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-03 05:53:31,368 WARN [Listener at localhost/37989] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-03 05:53:31,387 WARN [Listener at localhost/37989] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-03 05:53:31,395 INFO [Listener at localhost/37989] log.Slf4jLog(67): jetty-6.1.26 2023-04-03 05:53:31,414 INFO [Listener at localhost/37989] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/java.io.tmpdir/Jetty_localhost_41397_datanode____.tq0rad/webapp 2023-04-03 05:53:31,539 INFO [Listener at localhost/37989] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:41397 2023-04-03 05:53:31,997 WARN [Listener at localhost/37275] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-03 05:53:32,517 DEBUG [Listener at localhost/37275] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b 2023-04-03 05:53:32,523 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x510be4492cf3eb0c: Processing first storage report for DS-ac955f5c-316c-4346-88c1-503433793ca9 from datanode 483e758b-6075-4333-aa58-496ba5c29667 2023-04-03 05:53:32,525 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x510be4492cf3eb0c: from storage DS-ac955f5c-316c-4346-88c1-503433793ca9 node DatanodeRegistration(127.0.0.1:40553, datanodeUuid=483e758b-6075-4333-aa58-496ba5c29667, infoPort=41555, infoSecurePort=0, ipcPort=37275, storageInfo=lv=-57;cid=testClusterID;nsid=1930041253;c=1680501210117), blocks: 0, hasStaleStorage: true, processing time: 3 msecs, invalidatedBlocks: 0 2023-04-03 05:53:32,525 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x510be4492cf3eb0c: Processing first storage report for DS-6ae99835-e3d1-44a6-81b4-544c986fb848 from datanode 483e758b-6075-4333-aa58-496ba5c29667 2023-04-03 05:53:32,525 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x510be4492cf3eb0c: from storage DS-6ae99835-e3d1-44a6-81b4-544c986fb848 node DatanodeRegistration(127.0.0.1:40553, datanodeUuid=483e758b-6075-4333-aa58-496ba5c29667, infoPort=41555, infoSecurePort=0, ipcPort=37275, storageInfo=lv=-57;cid=testClusterID;nsid=1930041253;c=1680501210117), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-03 05:53:32,609 INFO [Listener at localhost/37275] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/cluster_8445de51-c43b-f77a-d5f2-488a4722dd0f/zookeeper_0, clientPort=63744, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/cluster_8445de51-c43b-f77a-d5f2-488a4722dd0f/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/cluster_8445de51-c43b-f77a-d5f2-488a4722dd0f/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-03 05:53:32,622 INFO [Listener at localhost/37275] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=63744 2023-04-03 05:53:32,629 INFO [Listener at localhost/37275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 05:53:32,632 INFO [Listener at localhost/37275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 05:53:33,317 INFO [Listener at localhost/37275] util.FSUtils(471): Created version file at hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa with version=8 2023-04-03 05:53:33,317 INFO [Listener at localhost/37275] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/hbase-staging 2023-04-03 05:53:33,640 INFO [Listener at localhost/37275] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-03 05:53:34,104 INFO [Listener at localhost/37275] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-03 05:53:34,138 INFO [Listener at localhost/37275] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-03 05:53:34,138 INFO [Listener at localhost/37275] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-03 05:53:34,139 INFO [Listener at localhost/37275] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-03 05:53:34,139 INFO [Listener at localhost/37275] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-03 05:53:34,139 INFO [Listener at localhost/37275] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-03 05:53:34,294 INFO [Listener at localhost/37275] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-03 05:53:34,371 DEBUG [Listener at localhost/37275] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-03 05:53:34,466 INFO [Listener at localhost/37275] ipc.NettyRpcServer(120): Bind to /172.31.14.131:33301 2023-04-03 05:53:34,477 INFO [Listener at localhost/37275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 05:53:34,479 INFO [Listener at localhost/37275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 05:53:34,498 INFO [Listener at localhost/37275] zookeeper.RecoverableZooKeeper(93): Process identifier=master:33301 connecting to ZooKeeper ensemble=127.0.0.1:63744 2023-04-03 05:53:34,539 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:333010x0, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-03 05:53:34,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:33301-0x10148956f760000 connected 2023-04-03 05:53:34,567 DEBUG [Listener at localhost/37275] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-03 05:53:34,568 DEBUG [Listener at localhost/37275] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-03 05:53:34,571 DEBUG [Listener at localhost/37275] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-03 05:53:34,579 DEBUG [Listener at localhost/37275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=33301 2023-04-03 05:53:34,579 DEBUG [Listener at localhost/37275] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=33301 2023-04-03 05:53:34,580 DEBUG [Listener at localhost/37275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=33301 2023-04-03 05:53:34,580 DEBUG [Listener at localhost/37275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=33301 2023-04-03 05:53:34,580 DEBUG [Listener at localhost/37275] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=33301 2023-04-03 05:53:34,586 INFO [Listener at localhost/37275] master.HMaster(444): hbase.rootdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa, hbase.cluster.distributed=false 2023-04-03 05:53:34,666 INFO [Listener at localhost/37275] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-03 05:53:34,666 INFO [Listener at localhost/37275] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-03 05:53:34,666 INFO [Listener at localhost/37275] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-03 05:53:34,667 INFO [Listener at localhost/37275] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-03 05:53:34,667 INFO [Listener at localhost/37275] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-03 05:53:34,667 INFO [Listener at localhost/37275] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-03 05:53:34,672 INFO [Listener at localhost/37275] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-03 05:53:34,674 INFO [Listener at localhost/37275] ipc.NettyRpcServer(120): Bind to /172.31.14.131:37397 2023-04-03 05:53:34,677 INFO [Listener at localhost/37275] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-03 05:53:34,684 DEBUG [Listener at localhost/37275] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-03 05:53:34,685 INFO [Listener at localhost/37275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 05:53:34,688 INFO [Listener at localhost/37275] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 05:53:34,690 INFO [Listener at localhost/37275] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:37397 connecting to ZooKeeper ensemble=127.0.0.1:63744 2023-04-03 05:53:34,694 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:373970x0, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-03 05:53:34,695 DEBUG [Listener at localhost/37275] zookeeper.ZKUtil(164): regionserver:373970x0, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-03 05:53:34,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:37397-0x10148956f760001 connected 2023-04-03 05:53:34,697 DEBUG [Listener at localhost/37275] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-03 05:53:34,698 DEBUG [Listener at localhost/37275] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-03 05:53:34,698 DEBUG [Listener at localhost/37275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=37397 2023-04-03 05:53:34,698 DEBUG [Listener at localhost/37275] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=37397 2023-04-03 05:53:34,699 DEBUG [Listener at localhost/37275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=37397 2023-04-03 05:53:34,700 DEBUG [Listener at localhost/37275] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=37397 2023-04-03 05:53:34,700 DEBUG [Listener at localhost/37275] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=37397 2023-04-03 05:53:34,702 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,33301,1680501213465 2023-04-03 05:53:34,714 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-03 05:53:34,715 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,33301,1680501213465 2023-04-03 05:53:34,740 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-03 05:53:34,740 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-03 05:53:34,740 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 05:53:34,741 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-03 05:53:34,743 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,33301,1680501213465 from backup master directory 2023-04-03 05:53:34,743 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-03 05:53:34,746 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,33301,1680501213465 2023-04-03 05:53:34,746 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-03 05:53:34,747 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-03 05:53:34,747 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,33301,1680501213465 2023-04-03 05:53:34,750 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-03 05:53:34,751 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-03 05:53:34,839 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/hbase.id with ID: 725639a8-d3fa-4069-ae02-f6a2aaf47e11 2023-04-03 05:53:34,881 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-03 05:53:34,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 05:53:34,937 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x219de320 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:34,977 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@646e1516, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:35,003 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 05:53:35,005 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-03 05:53:35,023 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-03 05:53:35,023 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-03 05:53:35,025 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:53:35,029 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:53:35,030 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-03 05:53:35,064 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/data/master/store-tmp 2023-04-03 05:53:35,095 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:53:35,095 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-03 05:53:35,096 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 05:53:35,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 05:53:35,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-03 05:53:35,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 05:53:35,096 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 05:53:35,096 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-03 05:53:35,098 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/WALs/jenkins-hbase4.apache.org,33301,1680501213465 2023-04-03 05:53:35,121 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C33301%2C1680501213465, suffix=, logDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/WALs/jenkins-hbase4.apache.org,33301,1680501213465, archiveDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/oldWALs, maxLogs=10 2023-04-03 05:53:35,179 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40553,DS-ac955f5c-316c-4346-88c1-503433793ca9,DISK] 2023-04-03 05:53:35,190 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:53:35,251 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/WALs/jenkins-hbase4.apache.org,33301,1680501213465/jenkins-hbase4.apache.org%2C33301%2C1680501213465.1680501215132 2023-04-03 05:53:35,252 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40553,DS-ac955f5c-316c-4346-88c1-503433793ca9,DISK]] 2023-04-03 05:53:35,253 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-03 05:53:35,253 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:53:35,256 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-03 05:53:35,258 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-03 05:53:35,313 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-03 05:53:35,319 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-03 05:53:35,342 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-03 05:53:35,353 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:53:35,359 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-03 05:53:35,361 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-03 05:53:35,375 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-03 05:53:35,379 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 05:53:35,380 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60965985, jitterRate=-0.0915360301733017}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-03 05:53:35,380 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-03 05:53:35,381 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-03 05:53:35,400 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-03 05:53:35,400 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-03 05:53:35,402 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-03 05:53:35,404 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-03 05:53:35,435 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 31 msec 2023-04-03 05:53:35,436 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-03 05:53:35,460 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-03 05:53:35,466 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-03 05:53:35,489 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-03 05:53:35,493 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-03 05:53:35,494 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-03 05:53:35,499 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-03 05:53:35,503 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-03 05:53:35,507 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 05:53:35,508 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-03 05:53:35,508 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-03 05:53:35,519 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-03 05:53:35,524 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-03 05:53:35,524 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-03 05:53:35,524 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 05:53:35,525 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,33301,1680501213465, sessionid=0x10148956f760000, setting cluster-up flag (Was=false) 2023-04-03 05:53:35,539 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 05:53:35,546 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-03 05:53:35,547 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,33301,1680501213465 2023-04-03 05:53:35,553 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 05:53:35,558 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-03 05:53:35,559 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,33301,1680501213465 2023-04-03 05:53:35,562 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.hbase-snapshot/.tmp 2023-04-03 05:53:35,606 INFO [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(952): ClusterId : 725639a8-d3fa-4069-ae02-f6a2aaf47e11 2023-04-03 05:53:35,611 DEBUG [RS:0;jenkins-hbase4:37397] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-03 05:53:35,622 DEBUG [RS:0;jenkins-hbase4:37397] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-03 05:53:35,622 DEBUG [RS:0;jenkins-hbase4:37397] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-03 05:53:35,627 DEBUG [RS:0;jenkins-hbase4:37397] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-03 05:53:35,628 DEBUG [RS:0;jenkins-hbase4:37397] zookeeper.ReadOnlyZKClient(139): Connect 0x0033bdb6 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:35,635 DEBUG [RS:0;jenkins-hbase4:37397] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ef453c0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:35,636 DEBUG [RS:0;jenkins-hbase4:37397] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@acbd6c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-03 05:53:35,669 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:37397 2023-04-03 05:53:35,674 INFO [RS:0;jenkins-hbase4:37397] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-03 05:53:35,674 INFO [RS:0;jenkins-hbase4:37397] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-03 05:53:35,674 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1023): About to register with Master. 2023-04-03 05:53:35,678 INFO [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,33301,1680501213465 with isa=jenkins-hbase4.apache.org/172.31.14.131:37397, startcode=1680501214665 2023-04-03 05:53:35,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-03 05:53:35,699 DEBUG [RS:0;jenkins-hbase4:37397] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-03 05:53:35,710 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-03 05:53:35,710 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-03 05:53:35,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-03 05:53:35,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-03 05:53:35,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-03 05:53:35,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 05:53:35,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-03 05:53:35,711 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 05:53:35,717 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680501245717 2023-04-03 05:53:35,719 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-03 05:53:35,722 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-03 05:53:35,722 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-03 05:53:35,732 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-03 05:53:35,734 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-03 05:53:35,741 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-03 05:53:35,741 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-03 05:53:35,742 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-03 05:53:35,742 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-03 05:53:35,743 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:35,745 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-03 05:53:35,748 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-03 05:53:35,748 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-03 05:53:35,752 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-03 05:53:35,752 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-03 05:53:35,755 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680501215754,5,FailOnTimeoutGroup] 2023-04-03 05:53:35,759 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680501215755,5,FailOnTimeoutGroup] 2023-04-03 05:53:35,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:35,759 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-03 05:53:35,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:35,761 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:35,768 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39231, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-03 05:53:35,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:35,806 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa 2023-04-03 05:53:35,806 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:37989 2023-04-03 05:53:35,806 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-03 05:53:35,811 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-03 05:53:35,812 DEBUG [RS:0;jenkins-hbase4:37397] zookeeper.ZKUtil(162): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:35,812 WARN [RS:0;jenkins-hbase4:37397] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-03 05:53:35,813 INFO [RS:0;jenkins-hbase4:37397] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-03 05:53:35,813 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1947): logDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/WALs/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:35,814 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,37397,1680501214665] 2023-04-03 05:53:35,823 DEBUG [RS:0;jenkins-hbase4:37397] zookeeper.ZKUtil(162): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:35,836 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-03 05:53:35,847 INFO [RS:0;jenkins-hbase4:37397] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-03 05:53:35,876 INFO [RS:0;jenkins-hbase4:37397] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-03 05:53:35,880 INFO [RS:0;jenkins-hbase4:37397] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-03 05:53:35,881 INFO [RS:0;jenkins-hbase4:37397] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:35,882 INFO [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-03 05:53:35,891 INFO [RS:0;jenkins-hbase4:37397] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:35,891 DEBUG [RS:0;jenkins-hbase4:37397] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 05:53:35,892 DEBUG [RS:0;jenkins-hbase4:37397] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 05:53:35,892 DEBUG [RS:0;jenkins-hbase4:37397] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 05:53:35,892 DEBUG [RS:0;jenkins-hbase4:37397] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 05:53:35,892 DEBUG [RS:0;jenkins-hbase4:37397] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 05:53:35,892 DEBUG [RS:0;jenkins-hbase4:37397] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-03 05:53:35,892 DEBUG [RS:0;jenkins-hbase4:37397] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 05:53:35,893 DEBUG [RS:0;jenkins-hbase4:37397] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 05:53:35,893 DEBUG [RS:0;jenkins-hbase4:37397] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 05:53:35,893 DEBUG [RS:0;jenkins-hbase4:37397] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-03 05:53:35,894 INFO [RS:0;jenkins-hbase4:37397] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:35,895 INFO [RS:0;jenkins-hbase4:37397] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:35,895 INFO [RS:0;jenkins-hbase4:37397] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:35,914 INFO [RS:0;jenkins-hbase4:37397] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-03 05:53:35,918 INFO [RS:0;jenkins-hbase4:37397] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,37397,1680501214665-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:35,941 INFO [RS:0;jenkins-hbase4:37397] regionserver.Replication(203): jenkins-hbase4.apache.org,37397,1680501214665 started 2023-04-03 05:53:35,941 INFO [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,37397,1680501214665, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:37397, sessionid=0x10148956f760001 2023-04-03 05:53:35,942 DEBUG [RS:0;jenkins-hbase4:37397] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-03 05:53:35,942 DEBUG [RS:0;jenkins-hbase4:37397] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:35,942 DEBUG [RS:0;jenkins-hbase4:37397] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37397,1680501214665' 2023-04-03 05:53:35,942 DEBUG [RS:0;jenkins-hbase4:37397] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:35,943 DEBUG [RS:0;jenkins-hbase4:37397] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:35,944 DEBUG [RS:0;jenkins-hbase4:37397] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-03 05:53:35,944 DEBUG [RS:0;jenkins-hbase4:37397] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-03 05:53:35,944 DEBUG [RS:0;jenkins-hbase4:37397] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:35,944 DEBUG [RS:0;jenkins-hbase4:37397] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,37397,1680501214665' 2023-04-03 05:53:35,944 DEBUG [RS:0;jenkins-hbase4:37397] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-03 05:53:35,945 DEBUG [RS:0;jenkins-hbase4:37397] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-03 05:53:35,945 DEBUG [RS:0;jenkins-hbase4:37397] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-03 05:53:35,945 INFO [RS:0;jenkins-hbase4:37397] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-03 05:53:35,945 INFO [RS:0;jenkins-hbase4:37397] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-03 05:53:36,061 INFO [RS:0;jenkins-hbase4:37397] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37397%2C1680501214665, suffix=, logDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/WALs/jenkins-hbase4.apache.org,37397,1680501214665, archiveDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/oldWALs, maxLogs=32 2023-04-03 05:53:36,086 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40553,DS-ac955f5c-316c-4346-88c1-503433793ca9,DISK] 2023-04-03 05:53:36,090 INFO [RS:0;jenkins-hbase4:37397] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/WALs/jenkins-hbase4.apache.org,37397,1680501214665/jenkins-hbase4.apache.org%2C37397%2C1680501214665.1680501216064 2023-04-03 05:53:36,091 DEBUG [RS:0;jenkins-hbase4:37397] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40553,DS-ac955f5c-316c-4346-88c1-503433793ca9,DISK]] 2023-04-03 05:53:36,184 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-03 05:53:36,186 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-03 05:53:36,186 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa 2023-04-03 05:53:36,606 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:53:36,608 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-03 05:53:36,611 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/info 2023-04-03 05:53:36,612 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-03 05:53:36,613 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:53:36,613 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-03 05:53:36,615 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/rep_barrier 2023-04-03 05:53:36,616 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-03 05:53:36,617 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:53:36,617 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-03 05:53:36,619 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/table 2023-04-03 05:53:36,620 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-03 05:53:36,621 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:53:36,622 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740 2023-04-03 05:53:36,623 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740 2023-04-03 05:53:36,627 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 05:53:36,629 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-03 05:53:36,633 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 05:53:36,634 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73976682, jitterRate=0.10233846306800842}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 05:53:36,634 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-03 05:53:36,634 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-03 05:53:36,634 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-03 05:53:36,634 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-03 05:53:36,634 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-03 05:53:36,634 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-03 05:53:36,635 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-03 05:53:36,635 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-03 05:53:36,641 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-03 05:53:36,641 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-03 05:53:36,650 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-03 05:53:36,662 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-03 05:53:36,664 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-03 05:53:36,816 DEBUG [jenkins-hbase4:33301] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-03 05:53:36,819 INFO [PEWorker-3] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37397,1680501214665, state=OPENING 2023-04-03 05:53:36,827 DEBUG [PEWorker-3] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-03 05:53:36,828 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 05:53:36,829 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-03 05:53:36,832 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:53:37,013 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:37,015 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-03 05:53:37,019 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47712, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-03 05:53:37,029 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-03 05:53:37,029 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-03 05:53:37,033 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C37397%2C1680501214665.meta, suffix=.meta, logDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/WALs/jenkins-hbase4.apache.org,37397,1680501214665, archiveDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/oldWALs, maxLogs=32 2023-04-03 05:53:37,049 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:40553,DS-ac955f5c-316c-4346-88c1-503433793ca9,DISK] 2023-04-03 05:53:37,053 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/WALs/jenkins-hbase4.apache.org,37397,1680501214665/jenkins-hbase4.apache.org%2C37397%2C1680501214665.meta.1680501217034.meta 2023-04-03 05:53:37,053 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:40553,DS-ac955f5c-316c-4346-88c1-503433793ca9,DISK]] 2023-04-03 05:53:37,054 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-03 05:53:37,055 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-03 05:53:37,072 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-03 05:53:37,076 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-03 05:53:37,082 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-03 05:53:37,082 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:53:37,082 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-03 05:53:37,082 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-03 05:53:37,085 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-03 05:53:37,087 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/info 2023-04-03 05:53:37,087 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/info 2023-04-03 05:53:37,088 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-03 05:53:37,089 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:53:37,089 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-03 05:53:37,091 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/rep_barrier 2023-04-03 05:53:37,091 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/rep_barrier 2023-04-03 05:53:37,091 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-03 05:53:37,092 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:53:37,092 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-03 05:53:37,094 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/table 2023-04-03 05:53:37,094 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/table 2023-04-03 05:53:37,094 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-03 05:53:37,095 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:53:37,096 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740 2023-04-03 05:53:37,099 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740 2023-04-03 05:53:37,103 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 05:53:37,105 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-03 05:53:37,107 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69682542, jitterRate=0.03835079073905945}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 05:53:37,107 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-03 05:53:37,117 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680501217006 2023-04-03 05:53:37,134 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-03 05:53:37,135 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-03 05:53:37,136 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,37397,1680501214665, state=OPEN 2023-04-03 05:53:37,140 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-03 05:53:37,140 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-03 05:53:37,144 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-03 05:53:37,145 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,37397,1680501214665 in 308 msec 2023-04-03 05:53:37,150 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-03 05:53:37,151 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 496 msec 2023-04-03 05:53:37,157 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5380 sec 2023-04-03 05:53:37,157 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680501217157, completionTime=-1 2023-04-03 05:53:37,157 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-03 05:53:37,158 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-03 05:53:37,222 DEBUG [hconnection-0x55d29e8f-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:53:37,226 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47716, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:53:37,246 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-03 05:53:37,246 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680501277246 2023-04-03 05:53:37,246 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680501337246 2023-04-03 05:53:37,246 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 88 msec 2023-04-03 05:53:37,274 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33301,1680501213465-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:37,275 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33301,1680501213465-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:37,275 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33301,1680501213465-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:37,277 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:33301, period=300000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:37,277 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-03 05:53:37,284 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-03 05:53:37,294 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-03 05:53:37,296 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-03 05:53:37,305 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-03 05:53:37,308 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 05:53:37,310 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 05:53:37,335 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:53:37,338 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c empty. 2023-04-03 05:53:37,339 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:53:37,339 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-03 05:53:37,762 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-03 05:53:37,764 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6f4c130726253b9eea5a0f54de82205c, NAME => 'hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp 2023-04-03 05:53:38,177 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:53:38,177 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 6f4c130726253b9eea5a0f54de82205c, disabling compactions & flushes 2023-04-03 05:53:38,177 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. 2023-04-03 05:53:38,177 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. 2023-04-03 05:53:38,177 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. after waiting 0 ms 2023-04-03 05:53:38,177 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. 2023-04-03 05:53:38,177 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. 2023-04-03 05:53:38,177 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 6f4c130726253b9eea5a0f54de82205c: 2023-04-03 05:53:38,182 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 05:53:38,197 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680501218185"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501218185"}]},"ts":"1680501218185"} 2023-04-03 05:53:38,224 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 05:53:38,226 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 05:53:38,230 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501218226"}]},"ts":"1680501218226"} 2023-04-03 05:53:38,235 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-03 05:53:38,243 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6f4c130726253b9eea5a0f54de82205c, ASSIGN}] 2023-04-03 05:53:38,246 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=6f4c130726253b9eea5a0f54de82205c, ASSIGN 2023-04-03 05:53:38,247 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=6f4c130726253b9eea5a0f54de82205c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37397,1680501214665; forceNewPlan=false, retain=false 2023-04-03 05:53:38,399 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=6f4c130726253b9eea5a0f54de82205c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:38,399 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680501218398"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501218398"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501218398"}]},"ts":"1680501218398"} 2023-04-03 05:53:38,405 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 6f4c130726253b9eea5a0f54de82205c, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:53:38,565 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. 2023-04-03 05:53:38,566 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6f4c130726253b9eea5a0f54de82205c, NAME => 'hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c.', STARTKEY => '', ENDKEY => ''} 2023-04-03 05:53:38,567 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:53:38,567 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:53:38,567 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:53:38,567 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:53:38,570 INFO [StoreOpener-6f4c130726253b9eea5a0f54de82205c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:53:38,572 DEBUG [StoreOpener-6f4c130726253b9eea5a0f54de82205c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c/info 2023-04-03 05:53:38,572 DEBUG [StoreOpener-6f4c130726253b9eea5a0f54de82205c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c/info 2023-04-03 05:53:38,573 INFO [StoreOpener-6f4c130726253b9eea5a0f54de82205c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6f4c130726253b9eea5a0f54de82205c columnFamilyName info 2023-04-03 05:53:38,574 INFO [StoreOpener-6f4c130726253b9eea5a0f54de82205c-1] regionserver.HStore(310): Store=6f4c130726253b9eea5a0f54de82205c/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:53:38,576 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:53:38,577 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:53:38,582 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:53:38,589 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 05:53:38,590 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6f4c130726253b9eea5a0f54de82205c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63722858, jitterRate=-0.05045542120933533}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-03 05:53:38,590 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6f4c130726253b9eea5a0f54de82205c: 2023-04-03 05:53:38,592 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c., pid=6, masterSystemTime=1680501218558 2023-04-03 05:53:38,597 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. 2023-04-03 05:53:38,597 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. 2023-04-03 05:53:38,598 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=6f4c130726253b9eea5a0f54de82205c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:38,599 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680501218598"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680501218598"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680501218598"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680501218598"}]},"ts":"1680501218598"} 2023-04-03 05:53:38,608 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-03 05:53:38,608 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 6f4c130726253b9eea5a0f54de82205c, server=jenkins-hbase4.apache.org,37397,1680501214665 in 198 msec 2023-04-03 05:53:38,613 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-03 05:53:38,613 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=6f4c130726253b9eea5a0f54de82205c, ASSIGN in 365 msec 2023-04-03 05:53:38,615 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 05:53:38,615 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501218615"}]},"ts":"1680501218615"} 2023-04-03 05:53:38,619 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-03 05:53:38,623 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 05:53:38,626 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3260 sec 2023-04-03 05:53:38,710 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-03 05:53:38,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-03 05:53:38,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 05:53:38,750 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-03 05:53:38,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-03 05:53:38,770 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 32 msec 2023-04-03 05:53:38,784 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-03 05:53:38,795 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-03 05:53:38,800 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 16 msec 2023-04-03 05:53:38,809 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-03 05:53:38,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-03 05:53:38,812 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.065sec 2023-04-03 05:53:38,814 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-03 05:53:38,815 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-03 05:53:38,816 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-03 05:53:38,817 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33301,1680501213465-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-03 05:53:38,817 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,33301,1680501213465-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-03 05:53:38,828 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-03 05:53:38,912 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x45b2cf6e to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:38,917 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1bc25e59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:38,931 DEBUG [hconnection-0x5d5e6c96-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:53:38,942 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47724, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:53:38,952 INFO [Listener at localhost/37275] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,33301,1680501213465 2023-04-03 05:53:38,972 INFO [Listener at localhost/37275] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=206, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=323, ProcessCount=169, AvailableMemoryMB=4461 2023-04-03 05:53:38,988 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 05:53:38,992 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 05:53:39,000 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 05:53:39,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 05:53:39,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 05:53:39,008 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 05:53:39,011 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 05:53:39,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-03 05:53:39,013 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:39,015 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c empty. 2023-04-03 05:53:39,017 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:39,017 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 05:53:39,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 05:53:39,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 05:53:39,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 05:53:39,435 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 05:53:39,438 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 6dcb9105330925690aad25e3fde3c83c, NAME => 'TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp 2023-04-03 05:53:39,450 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:53:39,450 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 6dcb9105330925690aad25e3fde3c83c, disabling compactions & flushes 2023-04-03 05:53:39,450 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:39,450 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:39,450 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. after waiting 0 ms 2023-04-03 05:53:39,450 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:39,450 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:39,450 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:39,455 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 05:53:39,456 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501219456"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501219456"}]},"ts":"1680501219456"} 2023-04-03 05:53:39,459 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 05:53:39,460 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 05:53:39,461 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501219461"}]},"ts":"1680501219461"} 2023-04-03 05:53:39,463 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 05:53:39,468 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dcb9105330925690aad25e3fde3c83c, ASSIGN}] 2023-04-03 05:53:39,469 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dcb9105330925690aad25e3fde3c83c, ASSIGN 2023-04-03 05:53:39,471 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dcb9105330925690aad25e3fde3c83c, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37397,1680501214665; forceNewPlan=false, retain=false 2023-04-03 05:53:39,622 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=6dcb9105330925690aad25e3fde3c83c, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:39,623 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501219622"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501219622"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501219622"}]},"ts":"1680501219622"} 2023-04-03 05:53:39,626 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:53:39,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 05:53:39,786 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:39,786 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 6dcb9105330925690aad25e3fde3c83c, NAME => 'TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.', STARTKEY => '', ENDKEY => ''} 2023-04-03 05:53:39,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:39,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:53:39,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:39,787 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:39,789 INFO [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:39,791 DEBUG [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A 2023-04-03 05:53:39,791 DEBUG [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A 2023-04-03 05:53:39,793 INFO [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:53:39,793 INFO [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6dcb9105330925690aad25e3fde3c83c columnFamilyName A 2023-04-03 05:53:39,794 INFO [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] regionserver.HStore(310): Store=6dcb9105330925690aad25e3fde3c83c/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:53:39,794 INFO [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:39,796 DEBUG [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B 2023-04-03 05:53:39,797 DEBUG [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B 2023-04-03 05:53:39,797 INFO [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:53:39,797 INFO [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6dcb9105330925690aad25e3fde3c83c columnFamilyName B 2023-04-03 05:53:39,798 INFO [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] regionserver.HStore(310): Store=6dcb9105330925690aad25e3fde3c83c/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:53:39,798 INFO [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:39,800 DEBUG [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C 2023-04-03 05:53:39,800 DEBUG [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C 2023-04-03 05:53:39,800 INFO [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:53:39,801 INFO [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 6dcb9105330925690aad25e3fde3c83c columnFamilyName C 2023-04-03 05:53:39,802 INFO [StoreOpener-6dcb9105330925690aad25e3fde3c83c-1] regionserver.HStore(310): Store=6dcb9105330925690aad25e3fde3c83c/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:53:39,802 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:39,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:39,806 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:39,809 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 05:53:39,815 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:39,820 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 05:53:39,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 6dcb9105330925690aad25e3fde3c83c; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71497080, jitterRate=0.06538951396942139}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 05:53:39,821 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:39,823 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., pid=11, masterSystemTime=1680501219780 2023-04-03 05:53:39,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:39,826 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:39,827 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=6dcb9105330925690aad25e3fde3c83c, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:39,827 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501219827"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680501219827"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680501219827"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680501219827"}]},"ts":"1680501219827"} 2023-04-03 05:53:39,834 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-03 05:53:39,834 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 in 204 msec 2023-04-03 05:53:39,837 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-03 05:53:39,838 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dcb9105330925690aad25e3fde3c83c, ASSIGN in 367 msec 2023-04-03 05:53:39,839 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 05:53:39,839 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501219839"}]},"ts":"1680501219839"} 2023-04-03 05:53:39,842 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 05:53:39,845 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 05:53:39,847 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 840 msec 2023-04-03 05:53:40,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-03 05:53:40,137 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-03 05:53:40,142 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x5d4f988c to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:40,147 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23162110, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:40,150 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:53:40,153 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47736, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:53:40,156 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 05:53:40,158 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55196, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 05:53:40,166 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x72ea4a65 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:40,171 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e82daee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:40,171 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x1645b6b0 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:40,178 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3e8196e0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:40,179 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x7b52aa42 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:40,184 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2e2005d0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:40,185 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x1710407d to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:40,189 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@57de6c0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:40,189 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x7bc6d729 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:40,193 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@75f5a5e6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:40,194 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x00375c9d to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:40,199 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@64804459, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:40,199 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x2ec2fae8 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:40,232 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@142b9819, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:40,232 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x27685d29 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:40,237 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fcbe28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:40,237 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x5979620a to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:53:40,241 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c9ded19, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:53:40,249 DEBUG [hconnection-0xce38a73-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:53:40,250 DEBUG [hconnection-0x2a12d69-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:53:40,250 DEBUG [hconnection-0x43a25080-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:53:40,250 DEBUG [hconnection-0x1330534b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:53:40,251 DEBUG [hconnection-0x1479edee-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:53:40,254 DEBUG [hconnection-0x3fdf621b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:53:40,254 DEBUG [hconnection-0x27e010-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:53:40,255 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47750, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:53:40,255 DEBUG [hconnection-0x21233b14-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:53:40,256 DEBUG [hconnection-0xb79970f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:53:40,260 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47746, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:53:40,260 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47760, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:53:40,261 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47758, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:53:40,261 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:53:40,262 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47782, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:53:40,270 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47770, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:53:40,270 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47768, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:53:40,272 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47792, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:53:40,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:40,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:40,352 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:40,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:40,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:40,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:40,358 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,358 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:40,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:40,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:40,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:40,363 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,363 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:40,365 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 05:53:40,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:40,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,372 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:40,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:40,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:40,382 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:40,382 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:40,385 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:40,385 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:40,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:40,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:40,386 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:40,386 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:40,394 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:40,394 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:40,395 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:40,395 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:40,395 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:40,396 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:40,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:40,408 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:40,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:40,409 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,410 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,410 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:40,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:40,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:40,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78321788[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:40,420 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:40,420 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,429 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,430 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,430 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:40,430 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:40,431 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:40,431 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:40,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:40,444 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:40,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:40,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:40,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:40,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:40,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:40,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:40,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:40,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:40,453 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:40,453 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:40,453 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:40,453 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:40,453 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:40,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:40,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:40,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:40,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:40,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:40,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,463 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:40,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:40,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:40,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:40,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:40,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:40,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:40,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:40,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:40,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:40,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:40,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,482 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:40,482 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:40,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501220500, max timestamp=1680501220521Num uniques 859; 2023-04-03 05:53:40,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:53:40,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680501220500, max timestamp=1680501220521Num uniques 859; 2023-04-03 05:53:40,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-03 05:53:40,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680501220500, max timestamp=1680501220521Num uniques 859; 2023-04-03 05:53:40,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-03 05:53:40,585 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:40,590 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/fe1efb84860b4668915a5cc5dea3bf98 2023-04-03 05:53:40,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:40,606 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:40,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:40,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:40,633 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:40,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:40,656 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:40,656 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:40,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,659 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,659 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:40,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:40,659 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,659 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:40,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:40,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:40,669 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:40,669 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:40,670 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:40,672 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:40,672 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:40,685 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:40,685 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:40,685 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:40,685 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:40,688 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:40,688 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:40,689 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,689 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:40,689 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:40,691 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:40,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:40,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501280672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501280679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,693 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,694 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:40,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:40,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:40,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:40,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501280681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:40,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:40,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:40,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dfea55e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:40,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:40,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:40,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501280691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,699 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:40,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501280693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,706 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:40,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:40,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:40,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:40,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:40,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:40,712 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:40,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:40,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:40,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:40,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:40,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:40,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:40,717 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:40,717 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:40,717 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:40,717 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:40,717 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:40,722 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,722 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:40,723 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:40,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:40,723 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:40,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:40,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:40,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,729 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:40,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:40,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:40,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:40,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:40,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:40,766 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:40,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:40,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:40,766 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:40,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,765 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:40,766 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/61f713fcc4034c8cb2b673c12cf73653 2023-04-03 05:53:40,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,767 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:40,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:40,841 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:40,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501280839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:40,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501280841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:40,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501280843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:40,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501280843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:40,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501280846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,866 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:40,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:40,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:40,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:40,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:40,890 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:40,890 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:40,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:40,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:40,893 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,893 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:40,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:40,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:40,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,897 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:40,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:40,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:40,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:40,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:40,900 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:40,900 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:40,901 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:40,902 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:40,902 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:40,902 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:40,902 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:40,902 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:40,903 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:40,906 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:40,907 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:40,913 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,913 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:40,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:40,915 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,915 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:40,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:40,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:40,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:40,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e0498ca[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:40,919 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:40,919 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,920 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:40,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:40,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:40,921 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,921 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:40,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:40,929 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:40,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:40,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:40,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:40,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:40,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:40,935 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:40,935 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:40,935 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:40,935 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:40,935 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:40,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:40,938 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:40,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:40,939 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:40,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:40,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,942 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:40,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,962 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:40,962 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:40,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:40,962 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:40,962 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:40,962 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:40,963 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:40,963 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,963 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:40,963 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:40,962 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:40,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:40,963 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:40,965 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:40,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:40,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:40,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:41,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501281046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501281049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501281051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,054 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501281051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501281054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,065 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:41,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:41,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:41,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:41,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:41,075 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:41,075 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:41,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:41,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:41,078 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,078 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:41,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:41,079 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:41,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,081 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,081 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:41,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:41,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:41,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:41,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:41,083 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:41,083 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:41,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:41,088 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:41,088 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:41,088 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:41,088 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:41,088 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:41,088 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:41,090 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:41,090 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:41,090 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,091 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:41,091 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:41,095 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,095 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,096 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,096 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:41,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e061d9c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:41,100 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:41,100 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,103 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:41,103 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:41,103 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:41,104 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,104 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:41,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:41,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:41,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:41,108 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:41,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:41,114 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:41,114 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:41,114 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:41,114 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:41,114 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:41,117 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,117 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:41,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,117 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,119 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,121 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,131 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:41,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:41,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:41,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:41,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:41,133 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:41,133 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,133 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,133 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,133 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,133 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:41,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:41,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:41,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:41,134 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:41,134 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:41,234 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:41,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:41,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:41,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:41,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:41,255 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:41,255 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:41,260 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/e82d73f169c949e79a62185dc86307f3 2023-04-03 05:53:41,262 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:41,262 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:41,263 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,263 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,265 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,265 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:41,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:41,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:41,266 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,266 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:41,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,268 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:41,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,269 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:41,270 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:41,270 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:41,270 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:41,270 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:41,272 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:41,273 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:41,273 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:41,273 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:41,273 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:41,273 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:41,273 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:41,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:41,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/fe1efb84860b4668915a5cc5dea3bf98 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/fe1efb84860b4668915a5cc5dea3bf98 2023-04-03 05:53:41,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,281 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,282 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,282 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:41,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:41,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76e0c783[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:41,285 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:41,285 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,287 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,287 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:41,287 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:41,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:41,287 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:41,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,291 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:41,291 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:41,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/fe1efb84860b4668915a5cc5dea3bf98, entries=150, sequenceid=12, filesize=11.6 K 2023-04-03 05:53:41,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/61f713fcc4034c8cb2b673c12cf73653 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/61f713fcc4034c8cb2b673c12cf73653 2023-04-03 05:53:41,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:41,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:41,300 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:41,300 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:41,300 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:41,300 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:41,300 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:41,302 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,302 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,307 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:41,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:41,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:41,309 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,314 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,321 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:41,322 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:41,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:41,322 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:41,322 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:41,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:41,322 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,322 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:41,324 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:41,324 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:41,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:41,325 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,325 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:41,326 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,326 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,329 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/61f713fcc4034c8cb2b673c12cf73653, entries=150, sequenceid=12, filesize=11.6 K 2023-04-03 05:53:41,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/e82d73f169c949e79a62185dc86307f3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/e82d73f169c949e79a62185dc86307f3 2023-04-03 05:53:41,345 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/e82d73f169c949e79a62185dc86307f3, entries=150, sequenceid=12, filesize=11.6 K 2023-04-03 05:53:41,348 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=135.63 KB/138880 for 6dcb9105330925690aad25e3fde3c83c in 983ms, sequenceid=12, compaction requested=false 2023-04-03 05:53:41,350 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-03 05:53:41,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:41,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:41,402 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680501220661, max timestamp=1680501221380Num uniques 1009; 2023-04-03 05:53:41,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-03 05:53:41,408 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=131.55 KB heapSize=376.05 KB 2023-04-03 05:53:41,413 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:41,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 05:53:41,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:41,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:53:41,414 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:41,414 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-03 05:53:41,426 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:41,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:41,429 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501281423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,430 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501281425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501281429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501281430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:41,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:41,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501281432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:41,458 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:41,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:41,459 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:41,459 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:41,459 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,459 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,461 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:41,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:41,461 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:41,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:41,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,463 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:41,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:41,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:41,468 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:41,468 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:41,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:41,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:41,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:41,483 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:41,483 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:41,483 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:41,487 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:41,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:41,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:41,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:41,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:41,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,496 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,500 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,501 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:41,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1edc4854[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:41,503 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:41,503 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,508 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,508 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:41,508 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:41,508 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:41,510 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=60.38 KB at sequenceid=55 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/8375e37082064e99b3c4306e8eb29de9 2023-04-03 05:53:41,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,511 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:41,511 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:41,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:41,512 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:41,512 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:41,512 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:41,512 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:41,512 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:41,512 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:41,512 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:41,517 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,517 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,517 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:41,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:41,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,519 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,523 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:41,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:41,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:41,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,530 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:41,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:41,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:41,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:41,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:41,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:41,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,532 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:41,532 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:41,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:41,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501281537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501281537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,555 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501281540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501281540, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,585 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=55 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/3466e3a4a518406795088acc0c6e5ae8 2023-04-03 05:53:41,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501281576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,634 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:41,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:41,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:41,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:41,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:41,653 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:41,654 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:41,655 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:41,655 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:41,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,658 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=57.75 KB at sequenceid=55 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/be91a306e15148e6a60e141d5b89ea22 2023-04-03 05:53:41,659 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:41,659 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:41,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,661 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,661 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:41,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:41,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:41,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:41,668 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:41,668 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:41,672 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:41,673 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:41,673 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:41,676 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:41,676 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/8375e37082064e99b3c4306e8eb29de9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/8375e37082064e99b3c4306e8eb29de9 2023-04-03 05:53:41,676 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:41,677 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:41,677 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:41,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:41,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:41,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:41,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:41,690 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/8375e37082064e99b3c4306e8eb29de9, entries=200, sequenceid=55, filesize=13.9 K 2023-04-03 05:53:41,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,692 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,693 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/3466e3a4a518406795088acc0c6e5ae8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/3466e3a4a518406795088acc0c6e5ae8 2023-04-03 05:53:41,693 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,693 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:41,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:41,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2424518d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:41,703 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:41,703 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,705 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,705 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:41,705 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:41,705 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:41,706 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:41,708 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,708 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,708 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:41,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:41,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,718 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/3466e3a4a518406795088acc0c6e5ae8, entries=150, sequenceid=55, filesize=11.6 K 2023-04-03 05:53:41,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/be91a306e15148e6a60e141d5b89ea22 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/be91a306e15148e6a60e141d5b89ea22 2023-04-03 05:53:41,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:41,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:41,741 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:41,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:41,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:41,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:41,741 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:41,744 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,744 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,744 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:41,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:41,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:41,745 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/be91a306e15148e6a60e141d5b89ea22, entries=150, sequenceid=55, filesize=11.6 K 2023-04-03 05:53:41,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,749 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,752 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~138.26 KB/141575, heapSize ~395.93 KB/405432, currentSize=46.96 KB/48090 for 6dcb9105330925690aad25e3fde3c83c in 344ms, sequenceid=55, compaction requested=false 2023-04-03 05:53:41,752 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:41,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,757 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:41,757 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:41,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:41,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:41,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:41,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:41,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:41,761 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:41,761 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:41,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:41,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:41,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:41,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:41,766 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 05:53:41,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:41,772 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:41,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:41,772 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:41,772 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:41,772 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:41,834 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/38d0d94e57d847329f9132381d2ca4b2 2023-04-03 05:53:41,842 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-03 05:53:41,844 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-03 05:53:41,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501281837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,856 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501281840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501281843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501281846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,858 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501281848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,861 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:41,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:41,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:41,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:41,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:41,898 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:41,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:41,901 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:41,901 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:41,901 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,901 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,903 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:41,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:41,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:41,903 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,904 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,904 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:41,904 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:41,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:41,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:41,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:41,906 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:41,906 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:41,908 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:41,915 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:41,915 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:41,915 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:41,915 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:41,915 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:41,915 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:41,916 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:41,916 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:41,916 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,916 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:41,916 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:41,920 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:41,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ae05268[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:41,924 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:41,924 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,926 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,926 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:41,926 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:41,926 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:41,926 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,926 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:41,929 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,929 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/6c874ce13a5b41839861d3391ee60ac5 2023-04-03 05:53:41,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:41,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:41,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:41,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:41,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:41,934 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:41,934 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:41,934 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:41,934 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:41,934 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:41,936 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,936 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:41,936 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:41,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:41,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:41,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,939 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:41,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,942 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:41,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:41,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,948 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:41,949 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:41,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:41,949 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:41,949 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,949 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,949 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:41,949 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:41,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:41,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:41,949 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:41,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:41,949 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,953 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:41,953 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:41,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:41,953 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:41,953 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:41,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501281959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501281959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501281960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501281961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,963 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:41,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501281962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:41,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=67 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/a6ff5256373b4c528ca840ed2f6d5400 2023-04-03 05:53:41,997 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/38d0d94e57d847329f9132381d2ca4b2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/38d0d94e57d847329f9132381d2ca4b2 2023-04-03 05:53:42,010 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/38d0d94e57d847329f9132381d2ca4b2, entries=150, sequenceid=67, filesize=11.6 K 2023-04-03 05:53:42,014 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/6c874ce13a5b41839861d3391ee60ac5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/6c874ce13a5b41839861d3391ee60ac5 2023-04-03 05:53:42,026 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/6c874ce13a5b41839861d3391ee60ac5, entries=150, sequenceid=67, filesize=11.6 K 2023-04-03 05:53:42,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/a6ff5256373b4c528ca840ed2f6d5400 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/a6ff5256373b4c528ca840ed2f6d5400 2023-04-03 05:53:42,042 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/a6ff5256373b4c528ca840ed2f6d5400, entries=150, sequenceid=67, filesize=11.6 K 2023-04-03 05:53:42,052 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6dcb9105330925690aad25e3fde3c83c in 286ms, sequenceid=67, compaction requested=true 2023-04-03 05:53:42,052 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:42,054 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:42,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:42,058 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:42,058 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:53:42,058 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:53:42,060 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:42,060 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:42,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:42,065 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:53:42,068 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/B is initiating minor compaction (all files) 2023-04-03 05:53:42,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:42,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:42,068 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/B in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:42,068 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:42,068 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:42,068 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/61f713fcc4034c8cb2b673c12cf73653, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/3466e3a4a518406795088acc0c6e5ae8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/6c874ce13a5b41839861d3391ee60ac5] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=34.8 K 2023-04-03 05:53:42,069 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:42,069 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:42,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:42,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:42,071 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37998 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:53:42,073 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,077 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/A is initiating minor compaction (all files) 2023-04-03 05:53:42,077 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 61f713fcc4034c8cb2b673c12cf73653, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680501220337 2023-04-03 05:53:42,073 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:42,077 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/A in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:42,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:42,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:42,077 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/fe1efb84860b4668915a5cc5dea3bf98, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/8375e37082064e99b3c4306e8eb29de9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/38d0d94e57d847329f9132381d2ca4b2] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=37.1 K 2023-04-03 05:53:42,078 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3466e3a4a518406795088acc0c6e5ae8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=55, earliestPutTs=1680501221393 2023-04-03 05:53:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:42,079 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6c874ce13a5b41839861d3391ee60ac5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680501221427 2023-04-03 05:53:42,078 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting fe1efb84860b4668915a5cc5dea3bf98, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680501220337 2023-04-03 05:53:42,083 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8375e37082064e99b3c4306e8eb29de9, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=55, earliestPutTs=1680501221389 2023-04-03 05:53:42,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:42,084 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 38d0d94e57d847329f9132381d2ca4b2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680501221427 2023-04-03 05:53:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:42,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:42,085 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:42,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:42,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:42,087 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:42,102 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:42,103 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:42,104 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:42,104 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:42,108 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:42,110 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:42,116 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 05:53:42,116 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:42,117 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:42,117 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:42,117 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:42,117 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:42,117 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:42,152 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#B#compaction#9 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:42,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:42,174 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:42,192 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#A#compaction#11 average throughput is 0.36 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:42,224 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501282211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501282202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501282214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501282225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501282219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,331 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501282328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501282330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501282328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501282340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501282344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,536 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501282536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501282537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,538 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501282538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501282544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501282554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,645 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/83a6d6fba6ff438085aa4c3ce00791b2 2023-04-03 05:53:42,671 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/9b9e8315b9034159afd47780b33448d2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/9b9e8315b9034159afd47780b33448d2 2023-04-03 05:53:42,683 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/3da0d3d1b7ae41de8f021dcd47fd2ba7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3da0d3d1b7ae41de8f021dcd47fd2ba7 2023-04-03 05:53:42,692 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 05:53:42,714 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/B of 6dcb9105330925690aad25e3fde3c83c into 9b9e8315b9034159afd47780b33448d2(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:42,714 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/A of 6dcb9105330925690aad25e3fde3c83c into 3da0d3d1b7ae41de8f021dcd47fd2ba7(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:42,714 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:42,714 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:42,715 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/B, priority=13, startTime=1680501222058; duration=0sec 2023-04-03 05:53:42,715 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/A, priority=13, startTime=1680501222057; duration=0sec 2023-04-03 05:53:42,715 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:42,715 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:42,715 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:53:42,726 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:53:42,726 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/C is initiating minor compaction (all files) 2023-04-03 05:53:42,726 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/C in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:42,727 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/e82d73f169c949e79a62185dc86307f3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/be91a306e15148e6a60e141d5b89ea22, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/a6ff5256373b4c528ca840ed2f6d5400] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=34.8 K 2023-04-03 05:53:42,728 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e82d73f169c949e79a62185dc86307f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680501220337 2023-04-03 05:53:42,734 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting be91a306e15148e6a60e141d5b89ea22, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=55, earliestPutTs=1680501221393 2023-04-03 05:53:42,735 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting a6ff5256373b4c528ca840ed2f6d5400, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680501221427 2023-04-03 05:53:42,784 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#C#compaction#13 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:42,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501282839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501282840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,850 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/598eb85158cb4d349bc5142321274d77 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/598eb85158cb4d349bc5142321274d77 2023-04-03 05:53:42,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501282842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,852 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501282849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:42,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501282864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:42,881 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/C of 6dcb9105330925690aad25e3fde3c83c into 598eb85158cb4d349bc5142321274d77(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:42,885 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:42,885 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/C, priority=13, startTime=1680501222060; duration=0sec 2023-04-03 05:53:42,885 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:43,099 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/de616e914eac41e796ef1f99d8eabb26 2023-04-03 05:53:43,149 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/41cf6db376554b4590d4cec0ccfceae2 2023-04-03 05:53:43,160 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/83a6d6fba6ff438085aa4c3ce00791b2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/83a6d6fba6ff438085aa4c3ce00791b2 2023-04-03 05:53:43,171 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/83a6d6fba6ff438085aa4c3ce00791b2, entries=150, sequenceid=87, filesize=11.6 K 2023-04-03 05:53:43,173 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/de616e914eac41e796ef1f99d8eabb26 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/de616e914eac41e796ef1f99d8eabb26 2023-04-03 05:53:43,194 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/de616e914eac41e796ef1f99d8eabb26, entries=150, sequenceid=87, filesize=11.6 K 2023-04-03 05:53:43,197 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/41cf6db376554b4590d4cec0ccfceae2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/41cf6db376554b4590d4cec0ccfceae2 2023-04-03 05:53:43,206 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/41cf6db376554b4590d4cec0ccfceae2, entries=150, sequenceid=87, filesize=11.6 K 2023-04-03 05:53:43,210 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 6dcb9105330925690aad25e3fde3c83c in 1094ms, sequenceid=87, compaction requested=false 2023-04-03 05:53:43,210 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:43,210 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:43,210 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:43,213 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:43,214 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,214 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:43,214 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:43,216 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:43,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a4c4068[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:43,221 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:43,221 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,223 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:43,223 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:43,223 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:43,223 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,223 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:43,225 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:43,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:43,229 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:43,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:43,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:43,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:43,230 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:43,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,234 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,234 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:43,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:43,235 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,243 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:43,243 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:43,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:43,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:43,243 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,244 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,244 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,244 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:43,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:43,244 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:43,244 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:43,244 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,244 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:43,249 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,249 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:43,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:43,349 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:43,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:43,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:43,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:43,360 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 05:53:43,360 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:43,360 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:43,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:43,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:43,361 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:43,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:43,361 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:43,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:43,361 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:43,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:43,365 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:43,365 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:43,365 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,366 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,367 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:43,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:43,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:43,368 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,368 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,368 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:43,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:43,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:43,372 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:43,372 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:43,372 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:43,372 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:43,383 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:43,384 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:43,384 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:43,384 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:43,384 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:43,384 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:43,384 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:43,384 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:43,384 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:43,384 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,385 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:43,385 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:43,401 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,401 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,401 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:43,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:43,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ae299d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:43,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:43,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,415 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,415 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:43,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501283390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:43,416 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:43,417 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/73c748a164c54c97a9f4f4b27ab82014 2023-04-03 05:53:43,420 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,420 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:43,420 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:43,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501283413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,427 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501283415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501283417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501283417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:43,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:43,432 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:43,432 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:43,432 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:43,432 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:43,432 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:43,435 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,435 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:43,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,435 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,436 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,439 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:43,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:43,443 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:43,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:43,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:43,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:43,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:43,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,444 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,444 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,444 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,444 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,444 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:43,444 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:43,444 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:43,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:43,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/3f25d7f8f1724db380977dbb1553ec9e 2023-04-03 05:53:43,519 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501283518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501283531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,538 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501283538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,556 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:43,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501283538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:43,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501283557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,562 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=106 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/83c110a8ec384d93aa06b6eb4a4cc927 2023-04-03 05:53:43,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:43,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:43,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:43,573 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:43,574 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:43,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:43,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:43,578 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,578 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,579 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/73c748a164c54c97a9f4f4b27ab82014 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/73c748a164c54c97a9f4f4b27ab82014 2023-04-03 05:53:43,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:43,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:43,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:43,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:43,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:43,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:43,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:43,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:43,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:43,589 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:43,589 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:43,589 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:43,590 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:43,590 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:43,590 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:43,590 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:43,590 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,590 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:43,590 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:43,591 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/73c748a164c54c97a9f4f4b27ab82014, entries=150, sequenceid=106, filesize=11.6 K 2023-04-03 05:53:43,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,594 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/3f25d7f8f1724db380977dbb1553ec9e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/3f25d7f8f1724db380977dbb1553ec9e 2023-04-03 05:53:43,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:43,597 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@80a8fc[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:43,597 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:43,597 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,599 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,599 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:43,599 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:43,599 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:43,599 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,599 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:43,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:43,601 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,605 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/3f25d7f8f1724db380977dbb1553ec9e, entries=150, sequenceid=106, filesize=11.6 K 2023-04-03 05:53:43,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:43,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:43,612 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:43,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:43,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:43,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:43,612 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:43,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:43,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/83c110a8ec384d93aa06b6eb4a4cc927 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/83c110a8ec384d93aa06b6eb4a4cc927 2023-04-03 05:53:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,617 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,619 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,622 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:43,623 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:43,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:43,623 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,623 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:43,623 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,626 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,626 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,627 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,627 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,627 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:43,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:43,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:43,628 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:43,628 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:43,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:43,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/83c110a8ec384d93aa06b6eb4a4cc927, entries=150, sequenceid=106, filesize=11.6 K 2023-04-03 05:53:43,637 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-03 05:53:43,637 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-03 05:53:43,638 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 6dcb9105330925690aad25e3fde3c83c in 278ms, sequenceid=106, compaction requested=true 2023-04-03 05:53:43,638 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:43,641 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:43,641 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:53:43,641 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:53:43,641 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:43,641 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:43,645 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:53:43,645 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:53:43,645 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/A is initiating minor compaction (all files) 2023-04-03 05:53:43,645 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/B is initiating minor compaction (all files) 2023-04-03 05:53:43,645 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/A in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:43,645 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/B in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:43,646 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/9b9e8315b9034159afd47780b33448d2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/de616e914eac41e796ef1f99d8eabb26, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/3f25d7f8f1724db380977dbb1553ec9e] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=34.9 K 2023-04-03 05:53:43,646 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3da0d3d1b7ae41de8f021dcd47fd2ba7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/83a6d6fba6ff438085aa4c3ce00791b2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/73c748a164c54c97a9f4f4b27ab82014] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=34.9 K 2023-04-03 05:53:43,646 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 9b9e8315b9034159afd47780b33448d2, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680501221427 2023-04-03 05:53:43,649 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 3da0d3d1b7ae41de8f021dcd47fd2ba7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680501221427 2023-04-03 05:53:43,650 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting de616e914eac41e796ef1f99d8eabb26, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680501221828 2023-04-03 05:53:43,655 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3f25d7f8f1724db380977dbb1553ec9e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680501222215 2023-04-03 05:53:43,656 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 83a6d6fba6ff438085aa4c3ce00791b2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680501221828 2023-04-03 05:53:43,661 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 73c748a164c54c97a9f4f4b27ab82014, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680501222215 2023-04-03 05:53:43,698 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#B#compaction#18 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:43,698 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#A#compaction#19 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:43,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:43,728 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 05:53:43,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:43,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:43,729 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:43,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:43,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:43,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:43,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:43,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:43,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:43,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:43,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:43,757 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:43,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:43,769 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:43,769 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:43,770 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,770 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,772 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:43,772 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:43,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:43,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,775 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,776 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:43,777 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:43,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501283765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:43,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:43,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:43,778 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:43,778 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:43,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:43,780 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501283768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:43,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:43,781 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:43,781 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:43,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501283775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,783 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:43,783 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:43,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:43,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:43,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:43,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:43,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501283778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501283781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,791 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,791 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/487df9cd051947bbacf9acb50f35c5e4 2023-04-03 05:53:43,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,793 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,793 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:43,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@87bc996[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:43,795 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:43,795 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,798 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:43,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:43,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:43,798 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/813182bb7b594b7cae410dd61364a27a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/813182bb7b594b7cae410dd61364a27a 2023-04-03 05:53:43,800 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,800 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:43,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:43,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:43,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:43,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:43,812 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:43,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:43,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:43,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:43,812 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:43,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:43,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:43,817 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,818 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/A of 6dcb9105330925690aad25e3fde3c83c into 813182bb7b594b7cae410dd61364a27a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,818 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:43,818 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/A, priority=13, startTime=1680501223641; duration=0sec 2023-04-03 05:53:43,818 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:43,818 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:53:43,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,819 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,823 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:53:43,823 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/C is initiating minor compaction (all files) 2023-04-03 05:53:43,823 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/C in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:43,823 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/598eb85158cb4d349bc5142321274d77, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/41cf6db376554b4590d4cec0ccfceae2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/83c110a8ec384d93aa06b6eb4a4cc927] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=34.9 K 2023-04-03 05:53:43,825 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,825 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,825 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:43,828 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 598eb85158cb4d349bc5142321274d77, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=67, earliestPutTs=1680501221427 2023-04-03 05:53:43,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:43,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:43,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:43,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:43,829 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:43,829 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 41cf6db376554b4590d4cec0ccfceae2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1680501221828 2023-04-03 05:53:43,829 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:43,828 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:43,825 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,830 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,830 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,830 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,830 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,830 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:43,829 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:43,830 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 83c110a8ec384d93aa06b6eb4a4cc927, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680501222215 2023-04-03 05:53:43,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:43,845 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/0e7de16c3f08498bbc5f4f58b656bbaa 2023-04-03 05:53:43,860 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#C#compaction#22 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:43,880 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501283879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501283882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501283884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:43,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501283885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501283891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/c71830570f134793a1a863b2f056bd4a 2023-04-03 05:53:43,922 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/39831166e8684a838b38a9c289feeadf as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/39831166e8684a838b38a9c289feeadf 2023-04-03 05:53:43,930 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:43,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:43,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/487df9cd051947bbacf9acb50f35c5e4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/487df9cd051947bbacf9acb50f35c5e4 2023-04-03 05:53:43,935 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:43,939 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/C of 6dcb9105330925690aad25e3fde3c83c into 39831166e8684a838b38a9c289feeadf(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:43,939 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:43,939 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/C, priority=13, startTime=1680501223641; duration=0sec 2023-04-03 05:53:43,939 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:43,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:43,952 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:43,952 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:43,952 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:43,953 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:43,953 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:43,954 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,954 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,956 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:43,956 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:43,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:43,957 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,957 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:43,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,963 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:43,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:43,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:43,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:43,964 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:43,964 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:43,964 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:43,965 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:43,966 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/487df9cd051947bbacf9acb50f35c5e4, entries=200, sequenceid=123, filesize=13.9 K 2023-04-03 05:53:43,968 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/0e7de16c3f08498bbc5f4f58b656bbaa as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/0e7de16c3f08498bbc5f4f58b656bbaa 2023-04-03 05:53:43,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:43,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:43,972 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:43,972 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:43,972 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:43,972 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:43,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:43,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:43,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:43,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:43,975 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,975 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,981 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,981 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:43,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:43,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:43,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ac97f1[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:43,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:43,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:43,992 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:43,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:43,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:43,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:43,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:43,997 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:43,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:43,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:43,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:43,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:44,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:44,001 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:44,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:44,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:44,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:44,001 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:44,004 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,004 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,004 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:44,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:44,005 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,011 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,014 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/0e7de16c3f08498bbc5f4f58b656bbaa, entries=150, sequenceid=123, filesize=11.6 K 2023-04-03 05:53:44,015 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:44,016 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:44,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:44,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:44,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:44,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/c71830570f134793a1a863b2f056bd4a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/c71830570f134793a1a863b2f056bd4a 2023-04-03 05:53:44,016 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:44,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:44,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:44,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:44,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,020 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:44,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,020 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:44,032 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/c71830570f134793a1a863b2f056bd4a, entries=150, sequenceid=123, filesize=11.6 K 2023-04-03 05:53:44,044 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for 6dcb9105330925690aad25e3fde3c83c in 315ms, sequenceid=123, compaction requested=false 2023-04-03 05:53:44,044 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:44,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:44,089 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 05:53:44,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:44,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:44,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:44,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:44,092 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:44,092 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:44,121 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:44,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:44,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:44,140 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501284132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501284137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501284140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501284140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501284144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:44,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:44,154 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:44,154 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:44,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:44,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:44,158 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,158 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,159 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:44,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:44,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:44,160 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,160 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:44,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,161 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:44,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,161 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:44,162 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:44,162 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:44,162 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:44,162 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:44,163 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:44,164 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:44,164 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:44,164 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:44,164 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:44,164 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:44,165 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,165 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:44,165 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:44,169 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,169 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,170 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,170 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:44,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67b9b6b8[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:44,172 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:44,172 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,175 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,175 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:44,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,176 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:44,176 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:44,176 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:44,177 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/9fb824fbe5bd4c4db84b8b0cea60213a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/9fb824fbe5bd4c4db84b8b0cea60213a 2023-04-03 05:53:44,179 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,179 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:44,179 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:44,179 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:44,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:44,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:44,185 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:44,185 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:44,185 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:44,185 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:44,185 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:44,187 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,187 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:44,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:44,188 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,192 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,193 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/B of 6dcb9105330925690aad25e3fde3c83c into 9fb824fbe5bd4c4db84b8b0cea60213a(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:44,193 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:44,193 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/B, priority=13, startTime=1680501223641; duration=0sec 2023-04-03 05:53:44,193 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,197 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:44,197 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:44,197 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:44,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:44,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:44,197 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,197 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:44,199 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:44,199 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,199 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,199 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,199 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,199 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:44,199 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:44,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:44,244 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501284243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501284244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,251 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501284249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501284251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501284254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,299 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:44,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:44,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:44,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:44,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:44,308 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:44,308 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:44,309 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:44,309 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:44,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,311 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,311 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:44,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:44,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:44,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,312 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,312 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:44,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:44,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:44,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:44,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:44,316 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:44,316 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:44,321 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:44,322 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:44,322 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:44,323 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:44,323 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:44,323 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:44,323 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:44,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:44,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:44,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:44,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:44,328 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:44,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:44,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@627b25fb[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:44,333 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:44,333 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,338 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:44,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:44,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:44,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:44,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:44,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:44,341 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:44,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:44,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:44,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:44,341 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:44,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,346 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,349 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,353 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,353 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,353 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,353 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:44,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:44,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:44,353 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,353 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,353 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,353 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,353 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:44,356 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:44,356 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:44,356 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:44,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:44,357 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:44,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:44,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501284448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501284448, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,456 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:44,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501284455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501284456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:44,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:44,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:44,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:44,463 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,463 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:44,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501284462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,463 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:44,469 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:44,469 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:44,469 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,469 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:44,472 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:44,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,473 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,473 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:44,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:44,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:44,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:44,477 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:44,477 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:44,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:44,482 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:44,482 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:44,482 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:44,482 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:44,488 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:44,488 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:44,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,489 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:44,489 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:44,491 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,491 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,491 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,491 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:44,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@341a775b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:44,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:44,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,495 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:44,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:44,496 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,496 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:44,497 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:44,497 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,497 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:44,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,497 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:44,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:44,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:44,504 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:44,504 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:44,504 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:44,504 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:44,504 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:44,507 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,507 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,507 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:44,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,508 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,509 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,512 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:44,512 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:44,512 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:44,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:44,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:44,512 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:44,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:44,512 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,513 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,513 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,513 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:44,513 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:44,513 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,513 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,513 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:44,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:44,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:44,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/ba09a0e9b53a4e59a3356a35533dfd58 2023-04-03 05:53:44,566 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/e3a5778da2a942cb84a6267f024efdb0 2023-04-03 05:53:44,591 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=143 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/e3789644da5b4eda914c7e822743f3e0 2023-04-03 05:53:44,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/ba09a0e9b53a4e59a3356a35533dfd58 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/ba09a0e9b53a4e59a3356a35533dfd58 2023-04-03 05:53:44,609 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/ba09a0e9b53a4e59a3356a35533dfd58, entries=200, sequenceid=143, filesize=14.1 K 2023-04-03 05:53:44,611 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/e3a5778da2a942cb84a6267f024efdb0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/e3a5778da2a942cb84a6267f024efdb0 2023-04-03 05:53:44,617 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:44,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:44,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:44,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:44,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:44,626 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:44,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:44,628 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:44,628 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:44,629 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,629 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,630 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/e3a5778da2a942cb84a6267f024efdb0, entries=150, sequenceid=143, filesize=11.8 K 2023-04-03 05:53:44,630 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:44,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:44,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:44,631 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:44,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,635 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/e3789644da5b4eda914c7e822743f3e0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/e3789644da5b4eda914c7e822743f3e0 2023-04-03 05:53:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:44,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:44,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:44,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:44,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:44,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:44,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:44,637 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:44,637 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:44,637 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:44,637 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:44,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:44,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,639 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:44,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:44,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@407d6d36[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:44,643 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:44,643 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,645 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:44,645 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:44,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:44,647 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/e3789644da5b4eda914c7e822743f3e0, entries=150, sequenceid=143, filesize=11.8 K 2023-04-03 05:53:44,650 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:44,650 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:44,650 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:44,651 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,651 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 6dcb9105330925690aad25e3fde3c83c in 562ms, sequenceid=143, compaction requested=true 2023-04-03 05:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,651 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,652 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:44,652 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:53:44,652 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:44,652 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:53:44,652 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:44,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,656 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40744 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:53:44,656 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36014 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:53:44,656 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/A is initiating minor compaction (all files) 2023-04-03 05:53:44,656 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/B is initiating minor compaction (all files) 2023-04-03 05:53:44,656 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/A in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:44,657 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/B in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:44,657 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/813182bb7b594b7cae410dd61364a27a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/487df9cd051947bbacf9acb50f35c5e4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/ba09a0e9b53a4e59a3356a35533dfd58] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=39.8 K 2023-04-03 05:53:44,657 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/9fb824fbe5bd4c4db84b8b0cea60213a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/0e7de16c3f08498bbc5f4f58b656bbaa, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/e3a5778da2a942cb84a6267f024efdb0] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=35.2 K 2023-04-03 05:53:44,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,657 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 813182bb7b594b7cae410dd61364a27a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680501222215 2023-04-03 05:53:44,659 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 9fb824fbe5bd4c4db84b8b0cea60213a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680501222215 2023-04-03 05:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,659 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 487df9cd051947bbacf9acb50f35c5e4, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1680501223402 2023-04-03 05:53:44,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,660 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0e7de16c3f08498bbc5f4f58b656bbaa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1680501223402 2023-04-03 05:53:44,661 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting ba09a0e9b53a4e59a3356a35533dfd58, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680501223765 2023-04-03 05:53:44,662 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e3a5778da2a942cb84a6267f024efdb0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680501223765 2023-04-03 05:53:44,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:44,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:44,663 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:44,663 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:44,663 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:44,663 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:44,663 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:44,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:44,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,666 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:44,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,671 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,671 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:44,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:44,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:44,671 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:44,671 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,678 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,678 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,671 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:44,678 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,678 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:44,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:44,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:44,672 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:44,678 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,679 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,678 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:44,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:44,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:44,702 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#B#compaction#27 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:44,715 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#A#compaction#28 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:44,762 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 05:53:44,762 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:44,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:44,766 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:44,766 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:44,766 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:44,766 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:44,766 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:44,779 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:44,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:44,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:44,791 WARN [DataStreamer for file /user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/deca8407e2dc42868cff033bce3634a7] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-04-03 05:53:44,792 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/deca8407e2dc42868cff033bce3634a7 2023-04-03 05:53:44,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:44,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:44,798 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:44,798 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:44,805 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:44,805 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:44,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,809 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:44,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:44,809 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,811 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,811 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:44,814 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501284807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501284810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501284810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501284810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:44,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:44,823 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:44,823 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:44,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:44,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:44,830 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501284814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,832 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:44,835 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:44,835 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:44,835 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:44,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:44,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:44,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,842 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,843 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,843 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:44,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a5b0ff7[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:44,847 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:44,847 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,849 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,849 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:44,856 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:44,857 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:44,857 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:44,860 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:44,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:44,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,878 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/8d41b636e74242448914deb63a356f9b 2023-04-03 05:53:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:44,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:44,880 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:44,880 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:44,880 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:44,880 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:44,880 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:44,886 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,886 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:44,886 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:44,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:44,887 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,891 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:44,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:44,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:44,895 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:44,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:44,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:44,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:44,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:44,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:44,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:44,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:44,896 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:44,896 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:44,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:44,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:44,917 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501284916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501284917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501284918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,927 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501284924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,933 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/2db6b11e318c453ab7a109bb71383c82 2023-04-03 05:53:44,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:44,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501284935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:44,948 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/deca8407e2dc42868cff033bce3634a7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/deca8407e2dc42868cff033bce3634a7 2023-04-03 05:53:44,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/deca8407e2dc42868cff033bce3634a7, entries=250, sequenceid=160, filesize=16.4 K 2023-04-03 05:53:44,969 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/8d41b636e74242448914deb63a356f9b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/8d41b636e74242448914deb63a356f9b 2023-04-03 05:53:44,979 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/8d41b636e74242448914deb63a356f9b, entries=150, sequenceid=160, filesize=11.8 K 2023-04-03 05:53:44,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/2db6b11e318c453ab7a109bb71383c82 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/2db6b11e318c453ab7a109bb71383c82 2023-04-03 05:53:44,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/2db6b11e318c453ab7a109bb71383c82, entries=150, sequenceid=160, filesize=11.8 K 2023-04-03 05:53:45,008 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:45,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:45,015 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 6dcb9105330925690aad25e3fde3c83c in 253ms, sequenceid=160, compaction requested=true 2023-04-03 05:53:45,015 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:45,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 05:53:45,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-03 05:53:45,016 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-03 05:53:45,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:45,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:45,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:45,032 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:45,032 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:45,033 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:45,033 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:45,034 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,034 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:45,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:45,037 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:45,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:45,038 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,038 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:45,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:45,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:45,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:45,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:45,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:45,039 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:45,039 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:45,041 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:45,052 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:45,052 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:45,056 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:45,056 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:45,056 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 05:53:45,057 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:45,058 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:45,059 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:45,059 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:45,059 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:45,059 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:45,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:45,124 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:45,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501285147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501285149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501285151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,157 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501285152, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501285150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,195 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/f701e37734c74514af2625013b3b4342 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/f701e37734c74514af2625013b3b4342 2023-04-03 05:53:45,196 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/4f6c156f926b4563b5f5f6d5352ad01a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/4f6c156f926b4563b5f5f6d5352ad01a 2023-04-03 05:53:45,207 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/A of 6dcb9105330925690aad25e3fde3c83c into f701e37734c74514af2625013b3b4342(size=12.1 K), total size for store is 28.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:45,207 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:45,207 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/A, priority=13, startTime=1680501224651; duration=0sec 2023-04-03 05:53:45,207 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=4), splitQueue=0 2023-04-03 05:53:45,207 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:53:45,209 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/B of 6dcb9105330925690aad25e3fde3c83c into 4f6c156f926b4563b5f5f6d5352ad01a(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:45,209 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:45,210 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/B, priority=13, startTime=1680501224652; duration=0sec 2023-04-03 05:53:45,210 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-04-03 05:53:45,211 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48050 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:53:45,211 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/C is initiating minor compaction (all files) 2023-04-03 05:53:45,211 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/C in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:45,211 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/39831166e8684a838b38a9c289feeadf, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/c71830570f134793a1a863b2f056bd4a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/e3789644da5b4eda914c7e822743f3e0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/2db6b11e318c453ab7a109bb71383c82] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=46.9 K 2023-04-03 05:53:45,212 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 39831166e8684a838b38a9c289feeadf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=106, earliestPutTs=1680501222215 2023-04-03 05:53:45,215 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 4 compacting, 0 eligible, 16 blocking 2023-04-03 05:53:45,216 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 05:53:45,216 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 05:53:45,216 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. because compaction request was cancelled 2023-04-03 05:53:45,216 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting c71830570f134793a1a863b2f056bd4a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1680501223402 2023-04-03 05:53:45,216 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-03 05:53:45,217 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e3789644da5b4eda914c7e822743f3e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680501223765 2023-04-03 05:53:45,217 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 05:53:45,217 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 05:53:45,217 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. because compaction request was cancelled 2023-04-03 05:53:45,218 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-03 05:53:45,218 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 2db6b11e318c453ab7a109bb71383c82, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680501224108 2023-04-03 05:53:45,219 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 05:53:45,219 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 05:53:45,219 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. because compaction request was cancelled 2023-04-03 05:53:45,240 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#C#compaction#33 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:45,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501285259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501285259, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501285262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501285263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501285264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501285463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501285463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501285466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501285466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501285469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,496 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/5eb94091ec7248e382483d1c9442f01b 2023-04-03 05:53:45,548 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/b87b5c9f6ccc4b09b38ec00400cade63 2023-04-03 05:53:45,616 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=176 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/6bebe12f735043fdb035e3cdcc34e1e5 2023-04-03 05:53:45,625 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/5eb94091ec7248e382483d1c9442f01b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5eb94091ec7248e382483d1c9442f01b 2023-04-03 05:53:45,637 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5eb94091ec7248e382483d1c9442f01b, entries=150, sequenceid=176, filesize=11.8 K 2023-04-03 05:53:45,641 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/b87b5c9f6ccc4b09b38ec00400cade63 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b87b5c9f6ccc4b09b38ec00400cade63 2023-04-03 05:53:45,655 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b87b5c9f6ccc4b09b38ec00400cade63, entries=150, sequenceid=176, filesize=11.8 K 2023-04-03 05:53:45,658 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/6bebe12f735043fdb035e3cdcc34e1e5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/6bebe12f735043fdb035e3cdcc34e1e5 2023-04-03 05:53:45,674 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/6bebe12f735043fdb035e3cdcc34e1e5, entries=150, sequenceid=176, filesize=11.8 K 2023-04-03 05:53:45,676 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/b155d984a46a4fa0bd87ecf630564c92 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/b155d984a46a4fa0bd87ecf630564c92 2023-04-03 05:53:45,677 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 6dcb9105330925690aad25e3fde3c83c in 621ms, sequenceid=176, compaction requested=true 2023-04-03 05:53:45,684 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:45,685 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:45,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,700 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:45,700 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:45,702 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:45,702 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5134bbba[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:45,705 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:45,705 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,707 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:45,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:45,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:45,708 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:45,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:45,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:45,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:45,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:45,711 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/C of 6dcb9105330925690aad25e3fde3c83c into b155d984a46a4fa0bd87ecf630564c92(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:45,711 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:45,711 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/C, priority=12, startTime=1680501225016; duration=0sec 2023-04-03 05:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:45,711 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:45,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:45,715 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:45,715 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:45,715 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:45,715 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:45,715 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:45,717 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,717 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:45,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,717 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:45,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:45,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:45,720 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:45,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,726 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:45,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:45,726 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:45,726 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,726 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:45,726 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:45,726 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,726 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:45,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:45,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:45,726 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,726 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:45,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:45,727 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:45,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:45,773 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 05:53:45,774 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:45,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:45,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:45,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:45,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:45,775 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:45,802 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501285789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501285795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501285799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501285802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501285804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,813 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/a32b57efdd14440a877f6e43394d0d7a 2023-04-03 05:53:45,827 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:45,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:45,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:45,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:45,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:45,838 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:45,838 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:45,852 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:45,852 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:45,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:45,855 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:45,855 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:45,856 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:45,856 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:45,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:45,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:45,857 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:45,857 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:45,857 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:45,857 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:45,863 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:45,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:45,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:45,872 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:45,872 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:45,872 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:45,872 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:45,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:45,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:45,875 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:45,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:45,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36f3ae68[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:45,878 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:45,878 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,880 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:45,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:45,881 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:45,881 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,881 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:45,882 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,883 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:45,883 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:45,883 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:45,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:45,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:45,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:45,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:45,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:45,899 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:45,899 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:45,899 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:45,899 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:45,899 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:45,907 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:45,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:45,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:45,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:45,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:45,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501285905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,910 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501285905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,910 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501285905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:45,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:45,912 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:45,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:45,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:45,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501285912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,914 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:45,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:45,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501285913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:45,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,919 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:45,919 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:45,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:45,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:45,920 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:45,920 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:45,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:45,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:45,920 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:45,920 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:45,924 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:45,924 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:45,924 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:45,924 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:45,924 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,925 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:45,925 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,024 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:46,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:46,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:46,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:46,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:46,030 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:46,030 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:46,030 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:46,030 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:46,031 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,031 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,032 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,032 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:46,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:46,033 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,033 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:46,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:46,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:46,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:46,034 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:46,034 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:46,034 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:46,034 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:46,035 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:46,035 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:46,035 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:46,035 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:46,035 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:46,035 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:46,035 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:46,035 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,035 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:46,035 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:46,036 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,036 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:46,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:46,037 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,037 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:46,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:46,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:46,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:46,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dcc715d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:46,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:46,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,045 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,045 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:46,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:46,046 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:46,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:46,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:46,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:46,050 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:46,050 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:46,050 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:46,050 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:46,050 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:46,052 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:46,052 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:46,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,052 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:46,052 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:46,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:46,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:46,053 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:46,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:46,055 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:46,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,065 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:46,065 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:46,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:46,065 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:46,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:46,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:46,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:46,065 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:46,065 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:46,066 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:46,066 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:46,066 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:46,066 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,066 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,066 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,066 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,066 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501286111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501286113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501286115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501286117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501286118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,166 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:46,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:46,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:46,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:46,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:46,188 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:46,188 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:46,188 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:46,188 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:46,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,190 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:46,190 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:46,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:46,191 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,191 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:46,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:46,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:46,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:46,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:46,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:46,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:46,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:46,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:46,192 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:46,193 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:46,193 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:46,193 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:46,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:46,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:46,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:46,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:46,194 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,194 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:46,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:46,195 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,195 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:46,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:46,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:46,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fdaf1a4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:46,198 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:46,198 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,199 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,199 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:46,199 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:46,199 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,199 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:46,199 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:46,206 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:46,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:46,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:46,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:46,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:46,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:46,209 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:46,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:46,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:46,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:46,210 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:46,211 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,212 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:46,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,212 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:46,212 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:46,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:46,214 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:46,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:46,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:46,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:46,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:46,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,223 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:46,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:46,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:46,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:46,223 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:46,223 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:46,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:46,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/ab5db2c70b2b4bd7a9ec7615834100b4 2023-04-03 05:53:46,282 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/bc9d3301927843bda621ae8689e2cad3 2023-04-03 05:53:46,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/a32b57efdd14440a877f6e43394d0d7a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/a32b57efdd14440a877f6e43394d0d7a 2023-04-03 05:53:46,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/a32b57efdd14440a877f6e43394d0d7a, entries=200, sequenceid=198, filesize=14.1 K 2023-04-03 05:53:46,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/ab5db2c70b2b4bd7a9ec7615834100b4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/ab5db2c70b2b4bd7a9ec7615834100b4 2023-04-03 05:53:46,308 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/ab5db2c70b2b4bd7a9ec7615834100b4, entries=150, sequenceid=198, filesize=11.8 K 2023-04-03 05:53:46,310 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/bc9d3301927843bda621ae8689e2cad3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/bc9d3301927843bda621ae8689e2cad3 2023-04-03 05:53:46,323 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:46,324 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/bc9d3301927843bda621ae8689e2cad3, entries=150, sequenceid=198, filesize=11.8 K 2023-04-03 05:53:46,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:46,327 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 6dcb9105330925690aad25e3fde3c83c in 554ms, sequenceid=198, compaction requested=true 2023-04-03 05:53:46,327 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:46,327 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:46,328 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:53:46,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:46,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:46,328 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:46,328 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:53:46,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:46,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:46,334 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:46,334 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:46,334 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:46,334 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:46,336 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,336 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,337 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48452 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:53:46,337 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/B is initiating minor compaction (all files) 2023-04-03 05:53:46,337 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/B in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:46,337 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/4f6c156f926b4563b5f5f6d5352ad01a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/8d41b636e74242448914deb63a356f9b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b87b5c9f6ccc4b09b38ec00400cade63, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/ab5db2c70b2b4bd7a9ec7615834100b4] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=47.3 K 2023-04-03 05:53:46,337 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 55622 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:53:46,338 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/A is initiating minor compaction (all files) 2023-04-03 05:53:46,338 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/A in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:46,338 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 4f6c156f926b4563b5f5f6d5352ad01a, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680501223765 2023-04-03 05:53:46,339 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/f701e37734c74514af2625013b3b4342, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/deca8407e2dc42868cff033bce3634a7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5eb94091ec7248e382483d1c9442f01b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/a32b57efdd14440a877f6e43394d0d7a] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=54.3 K 2023-04-03 05:53:46,339 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:46,339 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,341 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 8d41b636e74242448914deb63a356f9b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680501224108 2023-04-03 05:53:46,341 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting f701e37734c74514af2625013b3b4342, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=143, earliestPutTs=1680501223765 2023-04-03 05:53:46,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:46,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:46,341 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b87b5c9f6ccc4b09b38ec00400cade63, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680501224805 2023-04-03 05:53:46,342 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,342 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:46,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,342 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting deca8407e2dc42868cff033bce3634a7, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680501224108 2023-04-03 05:53:46,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:46,343 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 5eb94091ec7248e382483d1c9442f01b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680501224805 2023-04-03 05:53:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:46,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:46,343 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:46,343 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ab5db2c70b2b4bd7a9ec7615834100b4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680501225151 2023-04-03 05:53:46,343 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting a32b57efdd14440a877f6e43394d0d7a, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680501225151 2023-04-03 05:53:46,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:46,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:46,345 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:46,345 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:46,345 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:46,348 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:46,348 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:46,348 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:46,348 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:46,356 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:53:46,356 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:46,356 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:46,356 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:46,356 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:46,357 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:46,357 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:46,370 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#B#compaction#39 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:46,382 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#A#compaction#41 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:46,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:46,418 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:46,429 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/c64f73c360a64b2597a9f46fdf9bbc06 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/c64f73c360a64b2597a9f46fdf9bbc06 2023-04-03 05:53:46,430 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/acff5124d1f74aa8b442fb46260a94c2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/acff5124d1f74aa8b442fb46260a94c2 2023-04-03 05:53:46,436 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/e3cd9d3a6cbb4ed19b77cc43a6298f48 2023-04-03 05:53:46,442 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/B of 6dcb9105330925690aad25e3fde3c83c into c64f73c360a64b2597a9f46fdf9bbc06(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:46,442 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:46,442 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/B, priority=12, startTime=1680501226328; duration=0sec 2023-04-03 05:53:46,442 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:46,442 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:53:46,443 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/A of 6dcb9105330925690aad25e3fde3c83c into acff5124d1f74aa8b442fb46260a94c2(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:46,443 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:46,443 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/A, priority=12, startTime=1680501226328; duration=0sec 2023-04-03 05:53:46,443 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:46,444 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:53:46,444 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/C is initiating minor compaction (all files) 2023-04-03 05:53:46,444 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/C in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:46,444 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/b155d984a46a4fa0bd87ecf630564c92, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/6bebe12f735043fdb035e3cdcc34e1e5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/bc9d3301927843bda621ae8689e2cad3] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=35.6 K 2023-04-03 05:53:46,445 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b155d984a46a4fa0bd87ecf630564c92, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680501224108 2023-04-03 05:53:46,446 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6bebe12f735043fdb035e3cdcc34e1e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=176, earliestPutTs=1680501224805 2023-04-03 05:53:46,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,447 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting bc9d3301927843bda621ae8689e2cad3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680501225151 2023-04-03 05:53:46,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501286438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501286442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501286442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501286445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501286447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,458 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#C#compaction#43 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:46,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501286555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,557 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501286556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501286556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501286556, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501286557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501286759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501286760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501286760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501286760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:46,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501286766, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:46,876 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/62bbefade5e24051867a4875a93a3f60 2023-04-03 05:53:46,883 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/5a792de166e3409c8c2b7d2c837437df as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/5a792de166e3409c8c2b7d2c837437df 2023-04-03 05:53:46,896 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/C of 6dcb9105330925690aad25e3fde3c83c into 5a792de166e3409c8c2b7d2c837437df(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:46,896 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:46,896 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/C, priority=13, startTime=1680501226328; duration=0sec 2023-04-03 05:53:46,896 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:47,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501287064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501287065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,067 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501287065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501287067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501287070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,304 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/2e6c7cf1f31d4cf691b99aa0d139a4c5 2023-04-03 05:53:47,312 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/e3cd9d3a6cbb4ed19b77cc43a6298f48 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/e3cd9d3a6cbb4ed19b77cc43a6298f48 2023-04-03 05:53:47,323 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/e3cd9d3a6cbb4ed19b77cc43a6298f48, entries=150, sequenceid=212, filesize=11.8 K 2023-04-03 05:53:47,325 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/62bbefade5e24051867a4875a93a3f60 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/62bbefade5e24051867a4875a93a3f60 2023-04-03 05:53:47,335 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/62bbefade5e24051867a4875a93a3f60, entries=150, sequenceid=212, filesize=11.8 K 2023-04-03 05:53:47,336 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/2e6c7cf1f31d4cf691b99aa0d139a4c5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/2e6c7cf1f31d4cf691b99aa0d139a4c5 2023-04-03 05:53:47,343 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/2e6c7cf1f31d4cf691b99aa0d139a4c5, entries=150, sequenceid=212, filesize=11.8 K 2023-04-03 05:53:47,345 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6dcb9105330925690aad25e3fde3c83c in 996ms, sequenceid=212, compaction requested=false 2023-04-03 05:53:47,345 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:47,345 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:47,345 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:47,353 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:47,353 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,353 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:47,353 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:47,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:47,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:47,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:47,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:47,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:47,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b47668e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:47,358 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:47,358 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:47,360 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:47,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:47,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:47,360 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:47,360 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:47,360 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:47,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:47,363 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:47,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:47,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:47,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:47,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:47,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:47,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:47,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:47,371 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:47,371 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:47,371 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:47,371 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:47,371 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:47,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:47,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:47,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:47,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:47,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:47,376 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:47,380 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,383 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,383 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:47,383 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:47,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:47,384 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:47,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:47,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:47,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:47,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:47,384 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:47,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,384 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:47,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:47,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:47,385 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:47,385 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:47,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:47,484 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:47,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:47,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:47,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:47,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:47,489 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:47,489 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:47,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:47,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:47,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:47,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:47,492 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,492 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:47,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:47,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:47,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:47,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:47,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:47,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:47,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:47,497 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:47,497 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:47,497 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 05:53:47,497 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:47,497 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:47,497 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:47,497 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:47,497 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:47,497 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:47,521 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/52c1926fd7614c15ac66745b6e735d12 2023-04-03 05:53:47,553 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/969eec029adb40e09009550232aedb59 2023-04-03 05:53:47,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:47,573 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:47,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501287585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501287587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501287587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501287589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501287590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501287690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,693 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501287693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501287695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501287696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501287696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501287896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501287897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501287897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501287898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:47,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501287901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:47,989 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=233 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/38940e4819b9411e8f99ff36488b1170 2023-04-03 05:53:47,996 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/52c1926fd7614c15ac66745b6e735d12 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/52c1926fd7614c15ac66745b6e735d12 2023-04-03 05:53:48,003 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/52c1926fd7614c15ac66745b6e735d12, entries=150, sequenceid=233, filesize=11.8 K 2023-04-03 05:53:48,004 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/969eec029adb40e09009550232aedb59 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/969eec029adb40e09009550232aedb59 2023-04-03 05:53:48,010 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/969eec029adb40e09009550232aedb59, entries=150, sequenceid=233, filesize=11.8 K 2023-04-03 05:53:48,011 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/38940e4819b9411e8f99ff36488b1170 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/38940e4819b9411e8f99ff36488b1170 2023-04-03 05:53:48,017 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/38940e4819b9411e8f99ff36488b1170, entries=150, sequenceid=233, filesize=11.8 K 2023-04-03 05:53:48,019 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 6dcb9105330925690aad25e3fde3c83c in 521ms, sequenceid=233, compaction requested=true 2023-04-03 05:53:48,019 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:48,019 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:48,019 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:48,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:48,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:48,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:48,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:48,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70ec1360[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:48,025 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:48,025 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,028 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:48,028 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:48,028 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:48,028 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,028 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:48,030 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,030 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:48,030 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,030 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:48,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:48,034 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:48,034 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:48,034 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:48,034 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:48,034 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:48,036 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,036 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:48,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,036 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:48,036 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,038 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,045 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:48,045 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:48,045 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:48,045 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:48,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:48,045 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,045 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,045 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,045 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:48,045 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:48,045 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:48,045 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,046 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,046 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:48,146 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:48,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:48,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:48,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:48,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:48,157 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:48,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:48,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:48,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:48,158 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,158 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,159 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:48,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,160 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,160 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:48,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:48,176 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:48,176 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:48,176 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:48,188 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:48,188 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:48,188 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:48,188 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:48,188 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 05:53:48,188 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:48,189 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:48,189 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:48,189 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:48,189 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:48,190 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:48,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:48,203 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:48,205 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/157bf3f585454ad4adfea883597907f9 2023-04-03 05:53:48,226 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501288222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501288225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,232 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501288226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501288227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501288228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,257 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/b2763e9732174c86b02d5dc4251370e1 2023-04-03 05:53:48,291 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/f19487f5c1064ed9b7cd22ee43c987e5 2023-04-03 05:53:48,299 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/157bf3f585454ad4adfea883597907f9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/157bf3f585454ad4adfea883597907f9 2023-04-03 05:53:48,305 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/157bf3f585454ad4adfea883597907f9, entries=150, sequenceid=248, filesize=11.8 K 2023-04-03 05:53:48,307 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/b2763e9732174c86b02d5dc4251370e1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b2763e9732174c86b02d5dc4251370e1 2023-04-03 05:53:48,314 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b2763e9732174c86b02d5dc4251370e1, entries=150, sequenceid=248, filesize=11.8 K 2023-04-03 05:53:48,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/f19487f5c1064ed9b7cd22ee43c987e5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/f19487f5c1064ed9b7cd22ee43c987e5 2023-04-03 05:53:48,322 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/f19487f5c1064ed9b7cd22ee43c987e5, entries=150, sequenceid=248, filesize=11.8 K 2023-04-03 05:53:48,323 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 6dcb9105330925690aad25e3fde3c83c in 135ms, sequenceid=248, compaction requested=true 2023-04-03 05:53:48,323 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:48,323 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:48,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:48,324 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:48,324 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,324 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:48,324 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:48,326 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,326 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,327 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,327 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:48,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:48,331 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 05:53:48,331 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:48,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:48,331 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:48,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:48,331 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:48,331 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:48,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:48,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e54d2b9[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:48,334 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:48,334 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,336 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:48,336 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:48,336 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:48,336 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,336 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:48,339 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:48,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:48,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:48,344 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:48,344 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:48,344 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:48,344 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:48,344 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:48,344 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:48,348 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,348 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,348 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:48,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,349 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,350 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:48,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:48,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:48,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,362 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:48,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:48,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:48,363 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:48,363 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:48,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:48,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:48,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501288353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501288376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,419 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501288412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,446 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501288412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,447 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501288416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,463 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:48,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:48,466 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:48,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:48,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:48,468 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:48,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:48,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:48,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:48,469 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,469 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,471 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,471 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:48,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:48,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:48,471 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,471 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:48,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:48,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:48,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:48,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:48,476 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:48,476 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:48,476 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:48,476 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:48,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:48,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,480 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:48,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@767b5506[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:48,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:48,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:48,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:48,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:48,486 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,486 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:48,487 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:48,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:48,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:48,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:48,491 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:48,491 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:48,491 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:48,491 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:48,491 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:48,494 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,494 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,494 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,499 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,506 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:48,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:48,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:48,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,512 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:48,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:48,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:48,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:48,512 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:48,512 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:48,515 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501288514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501288521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501288548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,553 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501288548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,554 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501288550, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,612 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:48,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:48,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:48,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:48,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:48,617 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:48,617 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:48,617 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:48,617 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:48,618 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,618 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,619 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:48,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:48,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:48,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,624 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:48,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:48,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:48,628 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:48,637 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:48,637 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:48,638 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:48,638 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:48,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:48,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:48,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:48,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:48,640 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:48,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e83e51[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:48,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:48,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,644 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:48,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:48,644 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,644 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:48,647 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:48,647 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:48,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:48,652 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:48,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:48,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:48,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:48,653 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:48,655 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,655 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,655 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:48,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:48,656 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,659 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,667 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,667 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,667 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,667 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:48,667 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:48,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:48,667 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:48,667 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:48,667 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:48,667 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,668 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,668 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,667 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:48,668 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:48,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:48,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:48,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:48,720 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501288718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501288726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501288755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501288757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:48,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501288762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,767 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:48,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:48,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:48,771 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/a0c11360fb434b4081d96fd0d6d872bc 2023-04-03 05:53:48,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:48,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:48,772 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:48,772 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:48,772 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:48,772 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:48,773 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,773 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,775 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:48,776 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:48,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:48,777 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,777 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:48,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:48,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:48,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:48,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:48,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:48,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:48,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:48,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:48,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:48,787 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:48,787 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:48,787 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:48,787 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:48,789 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:48,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:48,805 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,805 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:48,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:48,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60f48ff[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:48,808 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:48,808 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,810 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,810 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:48,810 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:48,811 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:48,811 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,811 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:48,813 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:48,813 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:48,813 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:48,814 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:48,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:48,821 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:48,821 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:48,821 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:48,821 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:48,821 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:48,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:48,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:48,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,837 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:48,837 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:48,837 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,837 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,837 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:48,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:48,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:48,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:48,837 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:48,837 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:48,837 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,838 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:48,838 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:48,838 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,838 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:48,838 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:48,938 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:48,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:48,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:48,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:48,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:48,948 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:48,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:48,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:48,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:48,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,951 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,951 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:48,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:48,951 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,951 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:48,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:48,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:48,956 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:48,956 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:48,956 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:48,956 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:48,956 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:48,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:48,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:48,957 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:48,957 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:48,957 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:48,957 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:48,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:48,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:48,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:48,958 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:48,960 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,960 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:48,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:48,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e66939d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:48,971 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:48,971 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,972 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:48,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:48,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:48,973 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,973 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:48,975 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,975 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,976 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:48,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:48,980 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:48,980 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:48,980 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:48,980 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:48,980 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:48,983 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,983 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:48,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,983 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:48,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:48,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,986 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,989 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:48,989 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:48,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:48,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:48,989 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,989 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,989 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:48,989 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:48,989 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,989 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:48,989 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:48,990 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:48,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:48,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:48,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:48,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:48,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:48,993 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:48,993 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:49,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47768 deadline: 1680501289023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,034 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501289030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,059 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501289059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501289061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501289066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,093 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:49,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:49,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:49,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:49,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:49,097 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:49,097 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:49,097 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:49,097 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:49,098 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,098 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,099 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:49,099 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:49,099 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,100 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:49,100 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:49,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:49,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:49,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:49,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:49,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:49,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:49,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:49,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:49,101 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:49,101 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:49,101 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:49,101 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:49,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:49,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:49,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,102 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:49,102 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:49,104 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,104 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:49,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,105 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:49,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:49,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:49,107 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2193032[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:49,107 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:49,107 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,108 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,108 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,108 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:49,108 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:49,108 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:49,108 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:49,110 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:49,110 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,110 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:49,110 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:49,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:49,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:49,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:49,112 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:49,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:49,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:49,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:49,112 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:49,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:49,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,115 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:49,117 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:49,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:49,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:49,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,120 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:49,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:49,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:49,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:49,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:49,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:49,121 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:49,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:49,121 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:49,221 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/0f18767820084fb5ab9d2d5040815556 2023-04-03 05:53:49,221 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:49,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:49,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:49,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:49,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:49,228 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:49,229 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:49,229 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:49,229 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:49,236 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,236 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,238 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:49,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:49,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:49,239 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,239 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:49,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:49,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:49,254 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:49,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:49,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:49,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:49,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:49,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:49,255 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:49,255 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:49,255 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:49,255 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:49,255 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:49,257 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:49,259 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:49,261 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/986964e83bbf4a0b9afa8f9b456c5133 2023-04-03 05:53:49,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:49,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55d49c96[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:49,262 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:49,262 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,270 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:49,270 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:49,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:49,272 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/a0c11360fb434b4081d96fd0d6d872bc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/a0c11360fb434b4081d96fd0d6d872bc 2023-04-03 05:53:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:49,273 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:49,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:49,280 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:49,280 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:49,280 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:49,280 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:49,280 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:49,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/a0c11360fb434b4081d96fd0d6d872bc, entries=200, sequenceid=267, filesize=14.3 K 2023-04-03 05:53:49,283 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,283 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:49,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,283 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:49,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:49,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:49,284 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:49,285 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/0f18767820084fb5ab9d2d5040815556 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/0f18767820084fb5ab9d2d5040815556 2023-04-03 05:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:49,286 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:49,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:49,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:49,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,290 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:49,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:49,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:49,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:49,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:49,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:49,291 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:49,291 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:49,298 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/0f18767820084fb5ab9d2d5040815556, entries=150, sequenceid=267, filesize=11.9 K 2023-04-03 05:53:49,299 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/986964e83bbf4a0b9afa8f9b456c5133 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/986964e83bbf4a0b9afa8f9b456c5133 2023-04-03 05:53:49,316 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/986964e83bbf4a0b9afa8f9b456c5133, entries=150, sequenceid=267, filesize=11.9 K 2023-04-03 05:53:49,318 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=80.51 KB/82440 for 6dcb9105330925690aad25e3fde3c83c in 987ms, sequenceid=267, compaction requested=true 2023-04-03 05:53:49,318 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:49,318 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:49,318 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:53:49,318 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:53:49,318 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:49,319 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:49,321 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60774 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:53:49,321 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 63214 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:53:49,322 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/A is initiating minor compaction (all files) 2023-04-03 05:53:49,322 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/A in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:49,322 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/acff5124d1f74aa8b442fb46260a94c2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/e3cd9d3a6cbb4ed19b77cc43a6298f48, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/52c1926fd7614c15ac66745b6e735d12, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/157bf3f585454ad4adfea883597907f9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/a0c11360fb434b4081d96fd0d6d872bc] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=61.7 K 2023-04-03 05:53:49,321 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/B is initiating minor compaction (all files) 2023-04-03 05:53:49,322 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/B in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:49,322 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/c64f73c360a64b2597a9f46fdf9bbc06, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/62bbefade5e24051867a4875a93a3f60, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/969eec029adb40e09009550232aedb59, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b2763e9732174c86b02d5dc4251370e1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/0f18767820084fb5ab9d2d5040815556] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=59.3 K 2023-04-03 05:53:49,323 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting acff5124d1f74aa8b442fb46260a94c2, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680501225151 2023-04-03 05:53:49,325 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting c64f73c360a64b2597a9f46fdf9bbc06, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680501225151 2023-04-03 05:53:49,325 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e3cd9d3a6cbb4ed19b77cc43a6298f48, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1680501225788 2023-04-03 05:53:49,327 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 62bbefade5e24051867a4875a93a3f60, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1680501225788 2023-04-03 05:53:49,327 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 969eec029adb40e09009550232aedb59, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680501226441 2023-04-03 05:53:49,328 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b2763e9732174c86b02d5dc4251370e1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680501227585 2023-04-03 05:53:49,328 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 52c1926fd7614c15ac66745b6e735d12, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680501226441 2023-04-03 05:53:49,329 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0f18767820084fb5ab9d2d5040815556, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680501228223 2023-04-03 05:53:49,330 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 157bf3f585454ad4adfea883597907f9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680501227585 2023-04-03 05:53:49,330 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting a0c11360fb434b4081d96fd0d6d872bc, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680501228221 2023-04-03 05:53:49,350 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#B#compaction#54 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:49,351 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#A#compaction#55 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:49,389 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/7f2643d5f19a40dcac83fa7449bf0a55 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/7f2643d5f19a40dcac83fa7449bf0a55 2023-04-03 05:53:49,391 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:49,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:49,395 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/b42ab5f7d19f49b391ff22a0b0288cd0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b42ab5f7d19f49b391ff22a0b0288cd0 2023-04-03 05:53:49,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:49,399 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/A of 6dcb9105330925690aad25e3fde3c83c into 7f2643d5f19a40dcac83fa7449bf0a55(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:49,399 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:49,399 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/A, priority=11, startTime=1680501229318; duration=0sec 2023-04-03 05:53:49,399 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:49,399 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:53:49,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:49,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:49,400 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:49,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:49,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:49,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:49,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:49,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:49,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:49,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:49,404 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:49,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:49,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:49,405 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:49,406 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60774 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:53:49,406 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/C is initiating minor compaction (all files) 2023-04-03 05:53:49,406 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/C in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:49,406 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/5a792de166e3409c8c2b7d2c837437df, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/2e6c7cf1f31d4cf691b99aa0d139a4c5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/38940e4819b9411e8f99ff36488b1170, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/f19487f5c1064ed9b7cd22ee43c987e5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/986964e83bbf4a0b9afa8f9b456c5133] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=59.3 K 2023-04-03 05:53:49,407 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/B of 6dcb9105330925690aad25e3fde3c83c into b42ab5f7d19f49b391ff22a0b0288cd0(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:49,407 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:49,407 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/B, priority=11, startTime=1680501229318; duration=0sec 2023-04-03 05:53:49,407 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:49,408 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 5a792de166e3409c8c2b7d2c837437df, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=198, earliestPutTs=1680501225151 2023-04-03 05:53:49,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:49,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:49,417 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:49,417 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:49,418 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 2e6c7cf1f31d4cf691b99aa0d139a4c5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1680501225788 2023-04-03 05:53:49,418 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 38940e4819b9411e8f99ff36488b1170, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=233, earliestPutTs=1680501226441 2023-04-03 05:53:49,418 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 05:53:49,419 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:49,419 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting f19487f5c1064ed9b7cd22ee43c987e5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1680501227585 2023-04-03 05:53:49,419 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:49,419 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:49,419 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:49,420 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:49,420 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:49,420 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 986964e83bbf4a0b9afa8f9b456c5133, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680501228223 2023-04-03 05:53:49,443 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#C#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:49,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:49,532 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:49,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501289562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501289563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501289565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501289567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47746 deadline: 1680501289572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,670 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47792 deadline: 1680501289670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501289673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,857 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/5415e8c4796b487aa50a027ed6d96988 2023-04-03 05:53:49,871 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/8f9ae97794524f6e9c58729a3a3ccc14 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/8f9ae97794524f6e9c58729a3a3ccc14 2023-04-03 05:53:49,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501289878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:49,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501289883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,887 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/C of 6dcb9105330925690aad25e3fde3c83c into 8f9ae97794524f6e9c58729a3a3ccc14(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:49,887 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:49,887 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/C, priority=11, startTime=1680501229319; duration=0sec 2023-04-03 05:53:49,887 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:49,910 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/90ef1f21e18446738f280dfd528ab766 2023-04-03 05:53:49,949 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/068deae8a77c43baafc26ced11964da7 2023-04-03 05:53:49,958 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/5415e8c4796b487aa50a027ed6d96988 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5415e8c4796b487aa50a027ed6d96988 2023-04-03 05:53:49,965 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5415e8c4796b487aa50a027ed6d96988, entries=150, sequenceid=284, filesize=11.9 K 2023-04-03 05:53:49,966 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/90ef1f21e18446738f280dfd528ab766 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/90ef1f21e18446738f280dfd528ab766 2023-04-03 05:53:49,974 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/90ef1f21e18446738f280dfd528ab766, entries=150, sequenceid=284, filesize=11.9 K 2023-04-03 05:53:49,976 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/068deae8a77c43baafc26ced11964da7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/068deae8a77c43baafc26ced11964da7 2023-04-03 05:53:49,985 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/068deae8a77c43baafc26ced11964da7, entries=150, sequenceid=284, filesize=11.9 K 2023-04-03 05:53:49,986 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 6dcb9105330925690aad25e3fde3c83c in 568ms, sequenceid=284, compaction requested=false 2023-04-03 05:53:49,986 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:49,986 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:49,987 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:49,994 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:49,994 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,994 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:49,994 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:49,996 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:49,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:49,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:49,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:49,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:49,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:49,999 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29e07e63[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:49,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:49,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,000 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:50,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,001 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:50,001 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:50,001 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:50,003 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:50,003 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,003 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,003 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:50,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:50,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:50,009 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:50,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:50,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:50,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:50,009 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:50,012 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,012 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:50,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,012 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:50,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,015 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,019 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:50,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:50,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:50,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:50,020 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:50,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:50,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:50,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:50,020 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:50,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:50,121 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:50,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:50,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:50,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:50,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:50,128 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:50,128 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:50,128 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:50,128 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:50,131 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,131 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,132 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:50,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:50,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,134 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:50,134 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:50,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:50,134 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:50,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:50,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:50,135 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,135 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:50,135 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 05:53:50,135 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:50,135 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:50,136 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:50,136 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:50,136 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:50,136 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:50,176 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/209641050f1246769a9c28ab24d8f8b2 2023-04-03 05:53:50,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:50,189 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:50,216 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/5c3a2c238eb54836894badc279e733dc 2023-04-03 05:53:50,238 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/38330f764b414bd590588552c0f0bc0d 2023-04-03 05:53:50,245 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:50,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501290243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:50,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501290243, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,247 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/209641050f1246769a9c28ab24d8f8b2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/209641050f1246769a9c28ab24d8f8b2 2023-04-03 05:53:50,253 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/209641050f1246769a9c28ab24d8f8b2, entries=150, sequenceid=302, filesize=11.9 K 2023-04-03 05:53:50,254 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/5c3a2c238eb54836894badc279e733dc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/5c3a2c238eb54836894badc279e733dc 2023-04-03 05:53:50,260 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/5c3a2c238eb54836894badc279e733dc, entries=150, sequenceid=302, filesize=11.9 K 2023-04-03 05:53:50,261 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/38330f764b414bd590588552c0f0bc0d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/38330f764b414bd590588552c0f0bc0d 2023-04-03 05:53:50,268 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/38330f764b414bd590588552c0f0bc0d, entries=150, sequenceid=302, filesize=11.9 K 2023-04-03 05:53:50,272 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 6dcb9105330925690aad25e3fde3c83c in 137ms, sequenceid=302, compaction requested=true 2023-04-03 05:53:50,272 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:50,272 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,272 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:50,273 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:50,275 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,275 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,276 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:50,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:50,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3380eab9[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:50,278 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:50,278 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:50,280 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:50,281 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:50,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:50,284 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:50,284 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:50,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:50,287 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:50,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:50,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:50,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:50,287 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:50,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:50,290 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,292 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,297 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:50,297 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:50,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:50,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:50,297 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,297 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:50,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:50,297 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:50,297 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,298 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,298 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,298 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,298 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,298 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,298 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:50,298 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:50,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:50,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:50,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:50,348 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 05:53:50,348 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:50,349 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:50,349 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:50,349 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:50,349 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:50,349 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:50,358 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/c922b16b246046f4ba2bd56dc91d32f3 2023-04-03 05:53:50,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:50,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501290374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:50,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501290375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,398 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:50,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:50,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:50,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:50,402 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:50,402 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:50,402 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:50,402 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:50,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,405 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,405 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:50,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:50,406 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,406 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:50,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:50,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:50,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:50,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:50,407 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:50,408 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,408 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:50,408 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:50,408 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:50,410 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,410 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,410 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,410 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:50,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:50,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78d3d45b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:50,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:50,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,413 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,414 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:50,414 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:50,414 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:50,419 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:50,420 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,420 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:50,420 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:50,420 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:50,420 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:50,420 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:50,420 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:50,420 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:50,420 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:50,420 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:50,422 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,422 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,422 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:50,423 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,424 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,428 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:50,428 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,428 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:50,428 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:50,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:50,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:50,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:50,428 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,429 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:50,429 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,429 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:50,429 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,429 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,429 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,429 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:50,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:50,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:50,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:50,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501290480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,482 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:50,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501290480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,529 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:50,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:50,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:50,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:50,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:50,544 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:50,544 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:50,544 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:50,544 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:50,545 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,545 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,547 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:50,547 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:50,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:50,548 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,548 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:50,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:50,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,549 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:50,549 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:50,549 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:50,549 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:50,549 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:50,549 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:50,550 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,550 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:50,550 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:50,550 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:50,550 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:50,552 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,552 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,553 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,553 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:50,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:50,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@992d7fb[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:50,555 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:50,555 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,556 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:50,556 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,556 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:50,556 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:50,557 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:50,557 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,557 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,558 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:50,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:50,560 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:50,560 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:50,560 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:50,560 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:50,560 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:50,563 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,563 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,563 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,563 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,565 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,567 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,567 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:50,567 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:50,567 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,567 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:50,567 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:50,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:50,567 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,567 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,567 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:50,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:50,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:50,568 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:50,568 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:50,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:50,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:50,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:50,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501290575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:50,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47756 deadline: 1680501290575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:50,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47746 deadline: 1680501290578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,668 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:50,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:50,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:50,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:50,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:50,672 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:50,672 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:50,672 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:50,672 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:50,673 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,673 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,674 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,674 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:50,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:50,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:50,674 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,674 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:50,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:50,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:50,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:50,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:50,676 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:50,676 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:50,676 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:50,676 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:50,676 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:50,676 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,676 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:50,676 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:50,677 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:50,677 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:50,678 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,679 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,679 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:50,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@232bb3cc[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:50,680 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:50,680 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,683 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,683 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:50,683 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:50,683 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:50,683 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,683 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:50,684 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,684 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,684 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:50,684 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:50,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501290684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:50,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:50,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501290684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:50,687 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:50,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:50,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:50,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:50,687 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:50,691 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,691 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,691 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:50,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,698 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,698 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:50,698 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:50,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:50,698 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:50,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:50,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:50,698 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:50,698 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,698 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,699 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,699 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:50,699 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:50,699 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:50,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:50,787 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/af78dd718a654e8faff123e2f30b5a43 2023-04-03 05:53:50,799 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:50,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:50,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:50,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:50,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:50,812 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:50,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:50,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:50,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:50,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:50,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:50,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:50,815 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,815 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,815 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:50,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:50,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:50,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:50,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:50,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:50,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:50,828 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=318 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/dcad422310cc460ebec7c61897bf7a1a 2023-04-03 05:53:50,828 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,828 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:50,828 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:50,828 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:50,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:50,829 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,829 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:50,829 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:50,830 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,830 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,831 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:50,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17ca4032[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:50,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:50,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,834 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,835 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:50,835 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:50,835 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:50,835 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,835 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:50,836 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/c922b16b246046f4ba2bd56dc91d32f3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/c922b16b246046f4ba2bd56dc91d32f3 2023-04-03 05:53:50,840 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:50,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:50,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/c922b16b246046f4ba2bd56dc91d32f3, entries=200, sequenceid=318, filesize=14.3 K 2023-04-03 05:53:50,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:50,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:50,845 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:50,845 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:50,845 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:50,845 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:50,845 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:50,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/af78dd718a654e8faff123e2f30b5a43 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/af78dd718a654e8faff123e2f30b5a43 2023-04-03 05:53:50,852 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/af78dd718a654e8faff123e2f30b5a43, entries=150, sequenceid=318, filesize=11.9 K 2023-04-03 05:53:50,853 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,854 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:50,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,854 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:50,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:50,854 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/dcad422310cc460ebec7c61897bf7a1a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/dcad422310cc460ebec7c61897bf7a1a 2023-04-03 05:53:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,857 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:50,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,862 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/dcad422310cc460ebec7c61897bf7a1a, entries=150, sequenceid=318, filesize=11.9 K 2023-04-03 05:53:50,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:50,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:50,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:50,863 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:50,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:50,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:50,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:50,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:50,863 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:50,863 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:50,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:50,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:50,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:50,864 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 6dcb9105330925690aad25e3fde3c83c in 516ms, sequenceid=318, compaction requested=true 2023-04-03 05:53:50,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:50,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:50,864 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:50,864 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:53:50,864 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:53:50,864 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:50,865 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:50,866 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51798 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:53:50,866 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/A is initiating minor compaction (all files) 2023-04-03 05:53:50,866 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/A in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,866 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/7f2643d5f19a40dcac83fa7449bf0a55, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5415e8c4796b487aa50a027ed6d96988, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/209641050f1246769a9c28ab24d8f8b2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/c922b16b246046f4ba2bd56dc91d32f3] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=50.6 K 2023-04-03 05:53:50,866 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:53:50,867 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/B is initiating minor compaction (all files) 2023-04-03 05:53:50,867 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/B in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,867 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b42ab5f7d19f49b391ff22a0b0288cd0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/90ef1f21e18446738f280dfd528ab766, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/5c3a2c238eb54836894badc279e733dc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/af78dd718a654e8faff123e2f30b5a43] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=48.2 K 2023-04-03 05:53:50,867 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 7f2643d5f19a40dcac83fa7449bf0a55, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680501228223 2023-04-03 05:53:50,869 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b42ab5f7d19f49b391ff22a0b0288cd0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680501228223 2023-04-03 05:53:50,870 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 5415e8c4796b487aa50a027ed6d96988, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680501228340 2023-04-03 05:53:50,871 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 90ef1f21e18446738f280dfd528ab766, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680501228340 2023-04-03 05:53:50,871 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 209641050f1246769a9c28ab24d8f8b2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680501229554 2023-04-03 05:53:50,871 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 5c3a2c238eb54836894badc279e733dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680501229554 2023-04-03 05:53:50,871 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting c922b16b246046f4ba2bd56dc91d32f3, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680501230199 2023-04-03 05:53:50,872 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting af78dd718a654e8faff123e2f30b5a43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680501230199 2023-04-03 05:53:50,890 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#B#compaction#67 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:50,891 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#A#compaction#66 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:50,940 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/04b8fd4ab0394120bb222b06ea0bcc47 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/04b8fd4ab0394120bb222b06ea0bcc47 2023-04-03 05:53:50,952 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/B of 6dcb9105330925690aad25e3fde3c83c into 04b8fd4ab0394120bb222b06ea0bcc47(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:50,952 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:50,952 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/B, priority=12, startTime=1680501230864; duration=0sec 2023-04-03 05:53:50,952 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:50,953 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:53:50,955 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:53:50,955 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/C is initiating minor compaction (all files) 2023-04-03 05:53:50,955 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/C in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:50,955 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/8f9ae97794524f6e9c58729a3a3ccc14, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/068deae8a77c43baafc26ced11964da7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/38330f764b414bd590588552c0f0bc0d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/dcad422310cc460ebec7c61897bf7a1a] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=48.2 K 2023-04-03 05:53:50,956 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 8f9ae97794524f6e9c58729a3a3ccc14, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1680501228223 2023-04-03 05:53:50,957 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 068deae8a77c43baafc26ced11964da7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680501228340 2023-04-03 05:53:50,957 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/c4c5074970cb49f0b866b9ea41d5aae4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/c4c5074970cb49f0b866b9ea41d5aae4 2023-04-03 05:53:50,960 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 38330f764b414bd590588552c0f0bc0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680501229554 2023-04-03 05:53:50,961 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting dcad422310cc460ebec7c61897bf7a1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680501230199 2023-04-03 05:53:50,964 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:50,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:50,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:50,968 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/A of 6dcb9105330925690aad25e3fde3c83c into c4c5074970cb49f0b866b9ea41d5aae4(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:50,968 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:50,968 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/A, priority=12, startTime=1680501230864; duration=0sec 2023-04-03 05:53:50,968 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:50,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:50,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:50,998 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:50,998 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:50,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:50,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:51,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,002 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:51,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 05:53:51,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:51,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:51,004 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#C#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:51,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:51,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:51,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:51,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:51,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:51,005 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:51,005 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,005 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:51,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:51,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:51,007 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,007 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:51,007 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:51,007 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:51,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:51,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:51,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:51,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:51,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71dd3f9c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:51,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:51,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,015 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,016 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/7a905929bf0144f7bc98513a97578ab1 2023-04-03 05:53:51,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:51,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:51,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:51,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:51,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,019 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/0810c828ec6046429283ed59d22539b3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/0810c828ec6046429283ed59d22539b3 2023-04-03 05:53:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:51,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:51,023 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:51,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:51,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:51,024 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:51,024 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:51,030 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,030 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,030 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,032 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,033 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,036 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,036 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,036 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,036 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:51,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:51,036 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,036 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,036 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,036 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,036 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:51,036 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:51,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:51,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,039 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/C of 6dcb9105330925690aad25e3fde3c83c into 0810c828ec6046429283ed59d22539b3(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:51,039 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:51,039 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/C, priority=12, startTime=1680501230865; duration=0sec 2023-04-03 05:53:51,039 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:51,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:51,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501291055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:51,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501291056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,136 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:51,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:51,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:51,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:51,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:51,142 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:51,142 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,142 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:51,142 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:51,142 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,142 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,145 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,146 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,146 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:51,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:51,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:51,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:51,147 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:51,147 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,147 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:51,148 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,148 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:51,148 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:51,148 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:51,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:51,150 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:51,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6eef1353[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:51,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:51,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:51,153 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:51,154 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,154 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:51,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:51,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:51,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501291158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,159 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:51,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501291159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:51,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:51,163 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:51,163 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:51,163 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:51,163 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:51,163 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:51,166 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,167 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,167 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,168 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,169 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,173 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,173 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,173 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,173 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:51,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:51,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:51,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,173 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,173 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:51,174 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:51,174 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,174 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,174 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,174 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,174 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,274 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:51,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:51,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:51,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:51,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:51,278 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:51,278 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,278 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:51,278 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:51,279 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,279 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,280 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:51,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:51,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:51,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:51,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:51,282 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:51,282 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:51,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:51,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:51,283 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,283 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:51,283 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:51,283 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:51,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:51,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:51,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:51,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:51,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10247891[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:51,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:51,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,289 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:51,289 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:51,289 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:51,289 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,289 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:51,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:51,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:51,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:51,293 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:51,293 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:51,293 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:51,293 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:51,293 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:51,295 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,295 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,295 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,296 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,297 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,302 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,302 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,302 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,303 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:51,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:51,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,303 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:51,303 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:51,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:51,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501291360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:51,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501291362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,403 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:51,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:51,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:51,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:51,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:51,407 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:51,407 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,407 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:51,407 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:51,409 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,409 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,411 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:51,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:51,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:51,414 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:51,414 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:51,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:51,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:51,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:51,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:51,415 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,415 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:51,415 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:51,415 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:51,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:51,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:51,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:51,417 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:51,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55aa6177[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:51,419 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:51,419 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,420 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,420 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:51,420 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:51,420 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,420 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:51,420 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:51,422 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,423 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:51,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:51,425 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:51,425 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:51,425 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:51,425 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:51,425 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:51,427 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,427 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,427 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,428 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,429 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,432 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,432 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,432 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:51,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:51,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:51,432 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,432 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,432 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:51,432 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,432 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:51,432 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,433 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,433 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,433 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,447 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/1fe7ca60304a49739383b8b9e8b5f489 2023-04-03 05:53:51,533 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:51,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:51,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:51,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:51,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:51,536 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:51,536 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,536 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:51,536 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:51,536 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,536 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,538 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,539 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,539 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:51,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:51,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:51,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:51,541 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,541 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:51,541 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:51,541 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:51,541 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:51,544 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,544 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,544 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,544 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:51,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:51,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d139581[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:51,546 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:51,546 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,547 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:51,547 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:51,547 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,547 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:51,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,548 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,549 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:51,551 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:51,551 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:51,551 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:51,551 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:51,551 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:51,552 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,553 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,553 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,553 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,554 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,558 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,558 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,558 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,558 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,558 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:51,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:51,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:51,558 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,558 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,558 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,559 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:51,559 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:51,559 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,559 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,659 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:51,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:51,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:51,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:51,665 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:51,665 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,665 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:51,665 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:51,665 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,665 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:51,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501291664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,667 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:51,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:51,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501291666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:51,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,669 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:51,670 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:51,670 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,670 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:51,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:51,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:51,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:51,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:51,682 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,682 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,683 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:51,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b7072fb[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:51,684 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:51,684 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,687 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:51,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:51,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:51,691 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:51,691 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,691 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,691 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:51,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:51,694 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:51,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:51,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:51,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:51,694 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:51,696 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,696 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,696 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,697 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,700 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,700 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,700 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,700 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,700 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,700 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,700 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,700 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:51,701 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,700 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:51,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:51,701 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,701 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:51,701 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:51,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,801 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:51,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:51,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:51,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:51,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:51,807 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:51,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:51,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:51,808 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,808 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,818 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,818 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,819 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,819 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:51,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,820 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:51,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:51,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:51,820 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:51,820 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,821 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:51,828 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,828 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:51,828 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:51,828 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:51,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:51,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,831 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:51,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68a379f4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:51,834 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:51,834 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,837 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:51,838 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:51,838 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,838 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:51,840 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:51,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:51,848 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:51,848 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:51,848 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:51,848 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:51,848 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:51,851 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,851 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,851 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:51,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,852 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:51,857 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:51,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:51,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,860 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,860 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,860 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,861 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,861 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,861 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:51,861 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,861 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:51,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:51,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:51,861 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/d04718ce609248529ddb28c13108757a 2023-04-03 05:53:51,861 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:51,862 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:51,862 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:51,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:51,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:51,868 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/7a905929bf0144f7bc98513a97578ab1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/7a905929bf0144f7bc98513a97578ab1 2023-04-03 05:53:51,876 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/7a905929bf0144f7bc98513a97578ab1, entries=150, sequenceid=338, filesize=11.9 K 2023-04-03 05:53:51,878 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/1fe7ca60304a49739383b8b9e8b5f489 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/1fe7ca60304a49739383b8b9e8b5f489 2023-04-03 05:53:51,883 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/1fe7ca60304a49739383b8b9e8b5f489, entries=150, sequenceid=338, filesize=11.9 K 2023-04-03 05:53:51,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/d04718ce609248529ddb28c13108757a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/d04718ce609248529ddb28c13108757a 2023-04-03 05:53:51,897 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/d04718ce609248529ddb28c13108757a, entries=150, sequenceid=338, filesize=11.9 K 2023-04-03 05:53:51,898 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 6dcb9105330925690aad25e3fde3c83c in 895ms, sequenceid=338, compaction requested=false 2023-04-03 05:53:51,898 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:51,962 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:51,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:51,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:51,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:51,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:51,965 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:51,965 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,965 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:51,965 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:51,966 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,966 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:51,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:51,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:51,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:51,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:51,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:51,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:51,970 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:51,970 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:51,970 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:53:51,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:51,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:51,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:51,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:51,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:51,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:51,989 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/92ffe7c15a9b4034a4aaf5584cc10be7 2023-04-03 05:53:52,024 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/06d0204fe069437da3139990ab367081 2023-04-03 05:53:52,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:52,172 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:52,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501292227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501292227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,338 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501292335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501292335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,458 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=353 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/3a59e648c4da411d9bb8414179ae1b85 2023-04-03 05:53:52,465 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/92ffe7c15a9b4034a4aaf5584cc10be7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/92ffe7c15a9b4034a4aaf5584cc10be7 2023-04-03 05:53:52,470 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/92ffe7c15a9b4034a4aaf5584cc10be7, entries=150, sequenceid=353, filesize=11.9 K 2023-04-03 05:53:52,471 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/06d0204fe069437da3139990ab367081 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/06d0204fe069437da3139990ab367081 2023-04-03 05:53:52,477 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/06d0204fe069437da3139990ab367081, entries=150, sequenceid=353, filesize=11.9 K 2023-04-03 05:53:52,478 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/3a59e648c4da411d9bb8414179ae1b85 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/3a59e648c4da411d9bb8414179ae1b85 2023-04-03 05:53:52,484 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/3a59e648c4da411d9bb8414179ae1b85, entries=150, sequenceid=353, filesize=11.9 K 2023-04-03 05:53:52,485 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6dcb9105330925690aad25e3fde3c83c in 515ms, sequenceid=353, compaction requested=true 2023-04-03 05:53:52,485 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:52,485 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:52,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:52,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:52,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:52,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:52,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:52,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:52,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:52,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e041c4e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:52,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:52,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,492 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,493 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:52,493 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:52,493 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:52,493 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,493 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:52,494 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:52,494 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,494 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:52,494 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:52,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:52,497 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:52,497 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:52,498 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:52,498 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:52,498 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:52,499 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,499 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:52,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,499 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:52,500 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,502 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,504 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:52,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:52,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:52,504 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:52,504 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:52,504 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,505 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:52,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:52,505 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:52,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:52,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:52,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:52,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:52,542 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 05:53:52,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:52,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:52,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:52,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:52,543 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:52,543 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:52,563 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501292560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,564 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501292561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47756 deadline: 1680501292586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,589 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:53:52,594 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47746 deadline: 1680501292591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,595 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501292592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,596 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4183 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:53:52,597 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:53:52,605 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:52,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:52,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:52,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:52,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:52,608 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:52,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:52,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:52,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:52,609 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,609 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,616 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:52,616 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:52,617 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,617 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:52,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:52,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:52,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:52,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:52,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:52,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:52,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:52,618 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:52,618 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:52,618 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:52,619 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:52,619 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:52,619 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:52,619 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,619 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:52,619 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:52,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,620 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:52,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:52,621 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,621 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:52,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:52,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fea2392[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:52,622 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:52,623 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,624 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,624 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:52,624 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:52,624 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:52,624 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,624 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:52,625 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,626 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:52,626 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:52,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:52,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:52,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:52,628 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:52,628 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:52,628 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:52,628 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:52,628 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:52,631 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,631 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:52,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,631 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:52,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:52,632 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,635 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,635 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,635 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:52,635 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,635 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:52,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:52,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:52,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:52,635 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:52,636 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:52,635 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:52,636 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,636 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,636 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:52,636 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:52,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501292664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501292665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,736 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:52,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:52,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:52,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:52,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:52,739 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:52,739 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:52,739 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:52,739 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:52,740 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,740 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,741 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:52,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:52,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:52,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:52,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:52,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:52,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:52,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:52,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:52,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:52,743 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:52,743 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:52,743 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:52,743 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:52,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:52,745 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,745 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:52,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:52,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:52,752 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,752 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:52,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:52,753 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:52,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6755793e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:52,754 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:52,754 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,761 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:52,761 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,761 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:52,761 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:52,761 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:52,762 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:52,762 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:52,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:52,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:52,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:52,765 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:52,765 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:52,765 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:52,765 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:52,765 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:52,767 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,767 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,767 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:52,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:52,768 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,769 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:52,772 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:52,772 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:52,772 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:52,772 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:52,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:52,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:52,772 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,772 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,772 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,772 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,772 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:52,772 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:52,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:52,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:52,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501292866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501292868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,872 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:52,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:52,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:52,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:52,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:52,876 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:52,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:52,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:52,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:52,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,879 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:52,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:52,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,880 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:52,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:52,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:52,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:52,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:52,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:52,881 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:52,881 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:52,881 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:52,881 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:52,881 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:52,881 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:52,881 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:52,881 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:52,881 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,881 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:52,881 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:52,883 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,883 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:52,883 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:52,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:52,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14cad0f9[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:52,885 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:52,885 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,887 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,887 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,887 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:52,887 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:52,887 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:52,887 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:52,889 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:52,889 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,889 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:52,889 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:52,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:52,892 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:52,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:52,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:52,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:52,892 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:52,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:52,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:52,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:52,895 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,897 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:52,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:52,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:52,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:52,899 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:52,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:52,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:52,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,900 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:52,900 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:52,900 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:52,900 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:52,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:52,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:52,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:52,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:52,955 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/eab0428996754a57a9dd569513613466 2023-04-03 05:53:52,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/b5704d997ef04d2b93f103cf50f36080 2023-04-03 05:53:53,000 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:53,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:53,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:53,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:53,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:53,003 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:53,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:53,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:53,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:53,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,004 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,005 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:53,005 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:53,005 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,005 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,006 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,006 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:53,006 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:53,006 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:53,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:53,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:53,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:53,007 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:53,007 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:53,007 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:53,007 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:53,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,009 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:53,009 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,009 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ae25842[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:53,010 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:53,011 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,013 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:53,013 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:53,013 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:53,013 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:53,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:53,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:53,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:53,021 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:53,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:53,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:53,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:53,021 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:53,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:53,023 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:53,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:53,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,025 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,027 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:53,027 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:53,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:53,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:53,027 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,027 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:53,027 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,027 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:53,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,028 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:53,028 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:53,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:53,128 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:53,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:53,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:53,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:53,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:53,140 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:53,140 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:53,140 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:53,140 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:53,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:53,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:53,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:53,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:53,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:53,146 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:53,146 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:53,146 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:53,146 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:53,147 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:53,147 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:53,147 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:53,147 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,147 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:53,147 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:53,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,148 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,149 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:53,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:53,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:53,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34fa063f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:53,150 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:53,150 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,151 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:53,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:53,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:53,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:53,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:53,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:53,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:53,158 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:53,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:53,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:53,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:53,159 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:53,160 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,160 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:53,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,160 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:53,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:53,161 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,162 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:53,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:53,165 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:53,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:53,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:53,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:53,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:53,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,165 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:53,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,165 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:53,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:53,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:53,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501293170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:53,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501293170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,265 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:53,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:53,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:53,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:53,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:53,270 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:53,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:53,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:53,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:53,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,287 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:53,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:53,288 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,288 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:53,289 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:53,289 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:53,289 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:53,290 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:53,290 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:53,290 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:53,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:53,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:53,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:53,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:53,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,293 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:53,293 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:53,293 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@782a1517[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:53,295 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:53,295 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,296 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,296 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:53,296 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,296 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:53,296 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:53,296 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:53,298 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:53,298 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,298 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:53,298 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:53,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:53,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:53,301 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:53,301 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:53,301 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:53,301 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:53,301 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:53,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:53,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:53,303 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:53,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,305 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:53,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,310 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,310 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,310 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:53,310 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:53,310 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:53,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:53,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:53,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:53,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:53,310 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,310 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,311 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:53,311 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,311 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:53,311 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:53,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:53,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:53,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:53,381 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=372 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/b7b0644242974d0f8b07d0d4cba467fe 2023-04-03 05:53:53,388 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/eab0428996754a57a9dd569513613466 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/eab0428996754a57a9dd569513613466 2023-04-03 05:53:53,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/eab0428996754a57a9dd569513613466, entries=200, sequenceid=372, filesize=14.3 K 2023-04-03 05:53:53,394 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/b5704d997ef04d2b93f103cf50f36080 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b5704d997ef04d2b93f103cf50f36080 2023-04-03 05:53:53,400 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b5704d997ef04d2b93f103cf50f36080, entries=150, sequenceid=372, filesize=11.9 K 2023-04-03 05:53:53,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/b7b0644242974d0f8b07d0d4cba467fe as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/b7b0644242974d0f8b07d0d4cba467fe 2023-04-03 05:53:53,406 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/b7b0644242974d0f8b07d0d4cba467fe, entries=150, sequenceid=372, filesize=11.9 K 2023-04-03 05:53:53,407 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6dcb9105330925690aad25e3fde3c83c in 865ms, sequenceid=372, compaction requested=true 2023-04-03 05:53:53,407 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:53,407 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:53,407 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:53:53,407 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:53:53,407 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:53,408 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:53,409 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51934 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:53:53,410 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/A is initiating minor compaction (all files) 2023-04-03 05:53:53,410 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:53:53,410 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/A in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:53,410 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/B is initiating minor compaction (all files) 2023-04-03 05:53:53,410 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/B in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:53,410 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/c4c5074970cb49f0b866b9ea41d5aae4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/7a905929bf0144f7bc98513a97578ab1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/92ffe7c15a9b4034a4aaf5584cc10be7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/eab0428996754a57a9dd569513613466] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=50.7 K 2023-04-03 05:53:53,410 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/04b8fd4ab0394120bb222b06ea0bcc47, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/1fe7ca60304a49739383b8b9e8b5f489, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/06d0204fe069437da3139990ab367081, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b5704d997ef04d2b93f103cf50f36080] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=48.3 K 2023-04-03 05:53:53,410 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting c4c5074970cb49f0b866b9ea41d5aae4, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680501230199 2023-04-03 05:53:53,411 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 04b8fd4ab0394120bb222b06ea0bcc47, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680501230199 2023-04-03 05:53:53,411 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:53,411 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 7a905929bf0144f7bc98513a97578ab1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680501230367 2023-04-03 05:53:53,411 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1fe7ca60304a49739383b8b9e8b5f489, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680501230367 2023-04-03 05:53:53,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:53,411 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 92ffe7c15a9b4034a4aaf5584cc10be7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1680501231052 2023-04-03 05:53:53,412 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 06d0204fe069437da3139990ab367081, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1680501231052 2023-04-03 05:53:53,412 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b5704d997ef04d2b93f103cf50f36080, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680501232225 2023-04-03 05:53:53,412 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting eab0428996754a57a9dd569513613466, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680501232211 2023-04-03 05:53:53,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:53,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:53,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:53,416 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:53,416 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:53,416 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:53,416 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:53,417 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,417 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,418 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,419 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:53,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:53,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:53,419 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,419 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:53,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:53,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:53,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:53,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:53,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:53,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:53,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:53,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:53,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:53,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:53,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:53,421 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:53,421 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:53,422 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 05:53:53,422 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:53,422 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:53,422 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:53,422 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:53,422 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:53,422 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:53,424 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#A#compaction#78 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:53,425 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#B#compaction#79 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:53,447 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/02d9994e37fd44ba83e6a77379321bb1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/02d9994e37fd44ba83e6a77379321bb1 2023-04-03 05:53:53,456 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/B of 6dcb9105330925690aad25e3fde3c83c into 02d9994e37fd44ba83e6a77379321bb1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:53,456 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:53,456 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/B, priority=12, startTime=1680501233407; duration=0sec 2023-04-03 05:53:53,456 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:53,456 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:53:53,458 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:53:53,458 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/C is initiating minor compaction (all files) 2023-04-03 05:53:53,458 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/C in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:53,458 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/0810c828ec6046429283ed59d22539b3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/d04718ce609248529ddb28c13108757a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/3a59e648c4da411d9bb8414179ae1b85, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/b7b0644242974d0f8b07d0d4cba467fe] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=48.3 K 2023-04-03 05:53:53,458 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0810c828ec6046429283ed59d22539b3, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=318, earliestPutTs=1680501230199 2023-04-03 05:53:53,459 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting d04718ce609248529ddb28c13108757a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680501230367 2023-04-03 05:53:53,459 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3a59e648c4da411d9bb8414179ae1b85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=353, earliestPutTs=1680501231052 2023-04-03 05:53:53,459 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b7b0644242974d0f8b07d0d4cba467fe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680501232225 2023-04-03 05:53:53,468 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#C#compaction#81 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:53,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:53,679 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:53,714 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:53,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501293711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,717 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:53,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501293714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:53,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501293816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:53,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501293819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:53,845 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/bd7b1e836107469c8b364c0891e8318d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/bd7b1e836107469c8b364c0891e8318d 2023-04-03 05:53:53,851 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/439f27c4833b4c528faaf3652e7c42ab 2023-04-03 05:53:53,852 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/A of 6dcb9105330925690aad25e3fde3c83c into bd7b1e836107469c8b364c0891e8318d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:53,852 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:53,852 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/A, priority=12, startTime=1680501233407; duration=0sec 2023-04-03 05:53:53,853 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:53,882 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/a2ae1171ff0a4893ad74e9164a8ec622 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/a2ae1171ff0a4893ad74e9164a8ec622 2023-04-03 05:53:53,887 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/C of 6dcb9105330925690aad25e3fde3c83c into a2ae1171ff0a4893ad74e9164a8ec622(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:53,887 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:53,887 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/C, priority=12, startTime=1680501233408; duration=0sec 2023-04-03 05:53:53,888 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:54,019 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:54,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501294018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:54,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501294022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,269 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/70bbe05888524af28cbec2dbaa06c2bb 2023-04-03 05:53:54,311 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/32baea2932d44eb78dcae83769d6734d 2023-04-03 05:53:54,320 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/439f27c4833b4c528faaf3652e7c42ab as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/439f27c4833b4c528faaf3652e7c42ab 2023-04-03 05:53:54,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:54,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501294329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:54,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501294330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,335 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/439f27c4833b4c528faaf3652e7c42ab, entries=150, sequenceid=385, filesize=11.9 K 2023-04-03 05:53:54,336 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/70bbe05888524af28cbec2dbaa06c2bb as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/70bbe05888524af28cbec2dbaa06c2bb 2023-04-03 05:53:54,346 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/70bbe05888524af28cbec2dbaa06c2bb, entries=150, sequenceid=385, filesize=11.9 K 2023-04-03 05:53:54,347 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/32baea2932d44eb78dcae83769d6734d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/32baea2932d44eb78dcae83769d6734d 2023-04-03 05:53:54,355 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/32baea2932d44eb78dcae83769d6734d, entries=150, sequenceid=385, filesize=11.9 K 2023-04-03 05:53:54,356 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6dcb9105330925690aad25e3fde3c83c in 934ms, sequenceid=385, compaction requested=false 2023-04-03 05:53:54,356 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:54,356 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:54,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:54,358 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,358 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:54,358 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,358 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:54,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:54,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:54,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a254ff3[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:54,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:54,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,360 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:54,361 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:54,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:54,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:54,363 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:54,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:54,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:54,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:54,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:54,367 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:54,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:54,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:54,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:54,367 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:54,369 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:54,369 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:54,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:54,369 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:54,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:54,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:54,370 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:54,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:54,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:54,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:54,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:54,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,374 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:54,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:54,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:54,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:54,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:54,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:54,375 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:54,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:54,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:54,375 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:54,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:54,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:54,475 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:54,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:54,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:54,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:54,478 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:54,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:54,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:54,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:54,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:54,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:54,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:54,480 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:54,480 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,480 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:54,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:54,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:54,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:54,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:54,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:54,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:54,482 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:54,482 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:54,482 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:54,482 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:54,482 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:54,482 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:54,482 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 05:53:54,482 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:54,482 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:54,482 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:54,482 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:54,483 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:54,483 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:54,491 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/49913aa6f53b47d9b9df055a4ce481d4 2023-04-03 05:53:54,527 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/a70d734426414591ae45673da16f0fe4 2023-04-03 05:53:54,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:54,834 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:54,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:54,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501294855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:54,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501294856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,941 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=407 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/acf059be4a23422db8647e9e87f78309 2023-04-03 05:53:54,949 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/49913aa6f53b47d9b9df055a4ce481d4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/49913aa6f53b47d9b9df055a4ce481d4 2023-04-03 05:53:54,960 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/49913aa6f53b47d9b9df055a4ce481d4, entries=150, sequenceid=407, filesize=11.9 K 2023-04-03 05:53:54,962 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/a70d734426414591ae45673da16f0fe4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/a70d734426414591ae45673da16f0fe4 2023-04-03 05:53:54,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:54,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501294960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,963 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:54,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501294960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,968 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/a70d734426414591ae45673da16f0fe4, entries=150, sequenceid=407, filesize=11.9 K 2023-04-03 05:53:54,970 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/acf059be4a23422db8647e9e87f78309 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/acf059be4a23422db8647e9e87f78309 2023-04-03 05:53:54,981 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/acf059be4a23422db8647e9e87f78309, entries=150, sequenceid=407, filesize=11.9 K 2023-04-03 05:53:54,983 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6dcb9105330925690aad25e3fde3c83c in 501ms, sequenceid=407, compaction requested=true 2023-04-03 05:53:54,983 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:54,983 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:54,983 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:54,983 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:54,983 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:54,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:54,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,987 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:54,987 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,987 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:54,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:54,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:54,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:54,990 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72cd20bd[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:54,990 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:54,990 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,992 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:54,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:54,992 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,992 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:54,994 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:54,994 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:54,994 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:54,994 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:54,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:55,003 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:55,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:55,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:55,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:55,003 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:55,005 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,005 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:55,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,005 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:55,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:55,007 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,008 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:55,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:55,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:55,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:55,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:55,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:55,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:55,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:55,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:55,018 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:55,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:55,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:55,019 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:55,019 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:55,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:55,119 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:55,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:55,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:55,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:55,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:55,123 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:55,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:55,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:55,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:55,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:55,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:55,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:55,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:55,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:55,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:55,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:55,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:55,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:55,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:55,126 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:55,126 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:55,126 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:55,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:55,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:55,127 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:55,127 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:55,127 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 05:53:55,127 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:55,127 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:55,127 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:55,127 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:55,127 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:55,128 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:55,136 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/360ffaed2e2249958eb6510568249c31 2023-04-03 05:53:55,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:55,167 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:55,188 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:55,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501295185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:55,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501295188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,290 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:55,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501295289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:55,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501295291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,494 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:55,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501295492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,495 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:55,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501295494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,548 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/14b69eb9f122408680d9e61d926a5bd2 2023-04-03 05:53:55,564 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/9a3b6b8f405349bdba0e776886ce4ed8 2023-04-03 05:53:55,569 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/360ffaed2e2249958eb6510568249c31 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/360ffaed2e2249958eb6510568249c31 2023-04-03 05:53:55,574 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/360ffaed2e2249958eb6510568249c31, entries=100, sequenceid=420, filesize=9.5 K 2023-04-03 05:53:55,575 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/14b69eb9f122408680d9e61d926a5bd2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/14b69eb9f122408680d9e61d926a5bd2 2023-04-03 05:53:55,580 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/14b69eb9f122408680d9e61d926a5bd2, entries=100, sequenceid=420, filesize=9.5 K 2023-04-03 05:53:55,581 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/9a3b6b8f405349bdba0e776886ce4ed8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/9a3b6b8f405349bdba0e776886ce4ed8 2023-04-03 05:53:55,589 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/9a3b6b8f405349bdba0e776886ce4ed8, entries=100, sequenceid=420, filesize=9.5 K 2023-04-03 05:53:55,591 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 6dcb9105330925690aad25e3fde3c83c in 464ms, sequenceid=420, compaction requested=true 2023-04-03 05:53:55,592 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:55,592 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:55,592 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:55,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:55,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:55,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:55,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:55,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:55,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e20b25[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:55,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:55,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:55,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:55,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:55,599 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,599 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:55,601 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:55,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:55,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:55,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:55,604 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:55,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:55,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:55,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:55,604 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:55,606 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:55,607 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,607 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:55,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:55,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:55,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:55,610 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:55,612 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,612 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:55,612 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:55,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:55,612 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:55,613 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:55,612 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:55,613 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,613 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:55,613 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:55,613 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:55,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:55,613 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:55,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:55,613 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:55,613 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:55,713 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:55,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:55,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:55,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:55,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:55,725 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:55,725 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:55,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:55,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:55,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:55,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:55,727 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:55,728 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,728 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:55,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:55,729 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:55,730 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:55,730 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:55,730 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:55,730 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:55,731 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 05:53:55,731 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:55,731 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:55,731 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:55,731 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:55,731 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:55,731 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:55,741 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/9d3e1a84f3134341a82560078395a8dd 2023-04-03 05:53:55,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:55,799 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:55,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:55,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501295815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:55,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501295816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:55,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501295917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:55,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:55,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501295919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:56,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501296121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:56,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501296121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,160 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/fee978b8176443d9aff625d351fd9819 2023-04-03 05:53:56,171 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=439 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/bcf1b698d9b24c15b7788a5a65cf8239 2023-04-03 05:53:56,177 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/9d3e1a84f3134341a82560078395a8dd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/9d3e1a84f3134341a82560078395a8dd 2023-04-03 05:53:56,188 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/9d3e1a84f3134341a82560078395a8dd, entries=150, sequenceid=439, filesize=11.9 K 2023-04-03 05:53:56,189 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/fee978b8176443d9aff625d351fd9819 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/fee978b8176443d9aff625d351fd9819 2023-04-03 05:53:56,194 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/fee978b8176443d9aff625d351fd9819, entries=150, sequenceid=439, filesize=11.9 K 2023-04-03 05:53:56,195 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/bcf1b698d9b24c15b7788a5a65cf8239 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/bcf1b698d9b24c15b7788a5a65cf8239 2023-04-03 05:53:56,200 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/bcf1b698d9b24c15b7788a5a65cf8239, entries=150, sequenceid=439, filesize=11.9 K 2023-04-03 05:53:56,202 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 6dcb9105330925690aad25e3fde3c83c in 471ms, sequenceid=439, compaction requested=true 2023-04-03 05:53:56,202 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:56,202 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:56,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:56,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:56,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:56,204 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:56,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:56,207 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:56,207 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:56,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:56,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e482643[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:56,208 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:56,208 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:56,210 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:56,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:56,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:56,210 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:56,210 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:56,210 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:56,210 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,210 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:56,211 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:56,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:56,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:56,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:56,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:56,215 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:56,215 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:56,215 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:56,215 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:56,215 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:56,217 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:56,217 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:56,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:56,217 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:56,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:56,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:56,218 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,221 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:56,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:56,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:56,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:56,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:56,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:56,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:56,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:56,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:56,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:56,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:56,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:56,223 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:56,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:56,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:56,222 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:56,223 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:56,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:56,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:56,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:56,323 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:56,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:56,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:56,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:56,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:56,327 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:56,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:56,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:56,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:56,328 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:56,328 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:56,329 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,329 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:56,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:56,329 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,329 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:56,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:56,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:56,330 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:56,330 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:56,330 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:56,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:56,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:56,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:56,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:56,331 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:56,331 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:56,331 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 05:53:56,331 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:56,331 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:56,332 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:56,332 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:56,332 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:56,332 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:56,346 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/239f4ef397b645fa9fe3d26a4c0b646a 2023-04-03 05:53:56,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:56,425 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:56,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:56,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501296465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,467 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:56,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501296466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,570 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:56,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501296568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:56,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501296569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:56,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47746 deadline: 1680501296600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,604 DEBUG [Thread-143] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8190 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:53:56,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:56,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47756 deadline: 1680501296604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,606 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8193 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:53:56,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:56,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47750 deadline: 1680501296617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,621 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8204 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:53:56,758 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/96e4f3164e964a3b8f8447188b3d64fa 2023-04-03 05:53:56,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:56,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501296772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:56,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:56,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501296773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:57,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501297075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,075 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:57,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501297075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,185 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/ffec45ff61834421b6273449f9164f05 2023-04-03 05:53:57,191 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/239f4ef397b645fa9fe3d26a4c0b646a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/239f4ef397b645fa9fe3d26a4c0b646a 2023-04-03 05:53:57,198 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/239f4ef397b645fa9fe3d26a4c0b646a, entries=150, sequenceid=452, filesize=11.9 K 2023-04-03 05:53:57,199 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/96e4f3164e964a3b8f8447188b3d64fa as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/96e4f3164e964a3b8f8447188b3d64fa 2023-04-03 05:53:57,213 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/96e4f3164e964a3b8f8447188b3d64fa, entries=150, sequenceid=452, filesize=11.9 K 2023-04-03 05:53:57,219 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/ffec45ff61834421b6273449f9164f05 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/ffec45ff61834421b6273449f9164f05 2023-04-03 05:53:57,228 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/ffec45ff61834421b6273449f9164f05, entries=150, sequenceid=452, filesize=11.9 K 2023-04-03 05:53:57,229 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 6dcb9105330925690aad25e3fde3c83c in 898ms, sequenceid=452, compaction requested=true 2023-04-03 05:53:57,229 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:57,230 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:57,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:57,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:57,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:57,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:57,231 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:57,231 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:57,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:57,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:57,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:57,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e02acae[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:57,234 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:57,234 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:57,235 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:57,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:57,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:57,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:57,235 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,235 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:57,236 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:57,236 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:57,236 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,236 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:57,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:57,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:57,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:57,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:57,240 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:57,240 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:57,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:57,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:57,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:57,240 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:57,243 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:57,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:57,244 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:57,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:57,245 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:57,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:57,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:57,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:57,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:57,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:57,254 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,255 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:57,255 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:57,255 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:57,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:57,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:57,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:57,255 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,255 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:57,255 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:57,255 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:57,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:57,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:57,255 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:57,255 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:57,255 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:57,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:57,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:57,355 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:57,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:57,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:57,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:57,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:57,359 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:57,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:57,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:57,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:57,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:57,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:57,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,362 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:57,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:57,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:57,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:57,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:57,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:57,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:57,364 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:57,364 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:57,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:57,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:57,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:57,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:57,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:57,364 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:57,364 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:57,364 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 05:53:57,365 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:57,365 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:57,365 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:57,365 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:57,365 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:57,366 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:57,582 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:57,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:57,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:57,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501297617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,626 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:57,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501297623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:57,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501297724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:57,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501297727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,776 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/5518a9fb1dac4859abc08343dc8e8b19 2023-04-03 05:53:57,933 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:57,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501297930, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:57,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:57,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501297931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,193 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/8dee24a8c30e4a3dae2dc4ba91ff9110 2023-04-03 05:53:58,235 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:58,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501298234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:58,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501298237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,627 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/1e882e98ef184ecebe0e89cc29ef4595 2023-04-03 05:53:58,634 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/5518a9fb1dac4859abc08343dc8e8b19 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5518a9fb1dac4859abc08343dc8e8b19 2023-04-03 05:53:58,641 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5518a9fb1dac4859abc08343dc8e8b19, entries=150, sequenceid=472, filesize=11.9 K 2023-04-03 05:53:58,644 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/8dee24a8c30e4a3dae2dc4ba91ff9110 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/8dee24a8c30e4a3dae2dc4ba91ff9110 2023-04-03 05:53:58,655 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/8dee24a8c30e4a3dae2dc4ba91ff9110, entries=150, sequenceid=472, filesize=11.9 K 2023-04-03 05:53:58,657 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/1e882e98ef184ecebe0e89cc29ef4595 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/1e882e98ef184ecebe0e89cc29ef4595 2023-04-03 05:53:58,677 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/1e882e98ef184ecebe0e89cc29ef4595, entries=150, sequenceid=472, filesize=11.9 K 2023-04-03 05:53:58,678 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6dcb9105330925690aad25e3fde3c83c in 1314ms, sequenceid=472, compaction requested=true 2023-04-03 05:53:58,678 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:58,678 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:58,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:58,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:58,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:58,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:58,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,682 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:58,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:58,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:58,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3551a91b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:58,684 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:58,684 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,687 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:58,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:58,688 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,688 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:58,689 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:58,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:58,689 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:58,689 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:58,689 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:58,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:58,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:58,694 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:58,694 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:58,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:58,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:58,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:58,694 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:58,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:58,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,697 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:58,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:58,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:58,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,701 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:58,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:58,703 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:58,703 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:58,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:58,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:58,704 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,704 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:58,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:58,704 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:58,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:58,704 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:58,704 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,704 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,704 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,704 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:58,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:58,704 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:58,705 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:58,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:58,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:58,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 05:53:58,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:58,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:58,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:58,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:58,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:58,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:58,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/52debfab7b974bff8f648a0bd1861980 2023-04-03 05:53:58,782 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/bb0681573e1241c6aafec1e115d176e8 2023-04-03 05:53:58,788 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:58,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501298785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:58,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501298786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,797 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=486 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/348d8a0f4a654739bae07b1e6feea0ce 2023-04-03 05:53:58,802 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/52debfab7b974bff8f648a0bd1861980 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/52debfab7b974bff8f648a0bd1861980 2023-04-03 05:53:58,805 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:58,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:58,807 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/52debfab7b974bff8f648a0bd1861980, entries=250, sequenceid=486, filesize=16.7 K 2023-04-03 05:53:58,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:58,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:58,810 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/bb0681573e1241c6aafec1e115d176e8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/bb0681573e1241c6aafec1e115d176e8 2023-04-03 05:53:58,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:58,810 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:58,810 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:58,810 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:58,810 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:58,811 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,811 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:58,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:58,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:58,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:58,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:58,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:58,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:58,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:58,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:58,814 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:58,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:58,816 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/bb0681573e1241c6aafec1e115d176e8, entries=150, sequenceid=486, filesize=11.9 K 2023-04-03 05:53:58,816 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:58,816 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:58,817 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:58,817 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:58,817 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:58,817 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/348d8a0f4a654739bae07b1e6feea0ce as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/348d8a0f4a654739bae07b1e6feea0ce 2023-04-03 05:53:58,818 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,818 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:58,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,819 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:58,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:58,821 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33ec27b7[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:58,821 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:58,821 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,822 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,822 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:58,822 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:58,822 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:58,822 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,822 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:58,823 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/348d8a0f4a654739bae07b1e6feea0ce, entries=150, sequenceid=486, filesize=11.9 K 2023-04-03 05:53:58,824 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 6dcb9105330925690aad25e3fde3c83c in 82ms, sequenceid=486, compaction requested=true 2023-04-03 05:53:58,824 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:58,824 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:58,824 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-03 05:53:58,824 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-03 05:53:58,824 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:58,824 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:58,825 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:58,825 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:58,825 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,825 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:58,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,827 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 100810 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-03 05:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,827 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/A is initiating minor compaction (all files) 2023-04-03 05:53:58,827 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95930 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-03 05:53:58,827 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/A in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:58,827 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/B is initiating minor compaction (all files) 2023-04-03 05:53:58,827 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/B in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:58,827 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/bd7b1e836107469c8b364c0891e8318d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/439f27c4833b4c528faaf3652e7c42ab, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/49913aa6f53b47d9b9df055a4ce481d4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/360ffaed2e2249958eb6510568249c31, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/9d3e1a84f3134341a82560078395a8dd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/239f4ef397b645fa9fe3d26a4c0b646a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5518a9fb1dac4859abc08343dc8e8b19, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/52debfab7b974bff8f648a0bd1861980] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=98.4 K 2023-04-03 05:53:58,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:58,827 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/02d9994e37fd44ba83e6a77379321bb1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/70bbe05888524af28cbec2dbaa06c2bb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/a70d734426414591ae45673da16f0fe4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/14b69eb9f122408680d9e61d926a5bd2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/fee978b8176443d9aff625d351fd9819, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/96e4f3164e964a3b8f8447188b3d64fa, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/8dee24a8c30e4a3dae2dc4ba91ff9110, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/bb0681573e1241c6aafec1e115d176e8] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=93.7 K 2023-04-03 05:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,828 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting bd7b1e836107469c8b364c0891e8318d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680501232225 2023-04-03 05:53:58,828 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 02d9994e37fd44ba83e6a77379321bb1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680501232225 2023-04-03 05:53:58,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,828 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 439f27c4833b4c528faaf3652e7c42ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680501232555 2023-04-03 05:53:58,828 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 70bbe05888524af28cbec2dbaa06c2bb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680501232555 2023-04-03 05:53:58,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:58,829 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 49913aa6f53b47d9b9df055a4ce481d4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680501233709 2023-04-03 05:53:58,829 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a70d734426414591ae45673da16f0fe4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680501233709 2023-04-03 05:53:58,829 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:58,829 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:58,829 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:58,829 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:58,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:58,829 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:58,829 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 360ffaed2e2249958eb6510568249c31, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680501234837 2023-04-03 05:53:58,830 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 14b69eb9f122408680d9e61d926a5bd2, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680501234837 2023-04-03 05:53:58,830 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 9d3e1a84f3134341a82560078395a8dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680501235174 2023-04-03 05:53:58,831 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting fee978b8176443d9aff625d351fd9819, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680501235174 2023-04-03 05:53:58,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:58,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,831 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 96e4f3164e964a3b8f8447188b3d64fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680501235811 2023-04-03 05:53:58,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:58,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:58,831 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 239f4ef397b645fa9fe3d26a4c0b646a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680501235811 2023-04-03 05:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,832 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 8dee24a8c30e4a3dae2dc4ba91ff9110, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1680501236454 2023-04-03 05:53:58,832 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,832 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 5518a9fb1dac4859abc08343dc8e8b19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1680501236454 2023-04-03 05:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:58,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,833 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting bb0681573e1241c6aafec1e115d176e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680501237596 2023-04-03 05:53:58,833 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 52debfab7b974bff8f648a0bd1861980, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680501237596 2023-04-03 05:53:58,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:58,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,836 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,838 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,838 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:58,838 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:58,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:58,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:58,838 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:58,839 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,838 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:58,839 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:58,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:58,839 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,839 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,839 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:58,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:58,839 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:58,839 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:58,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:58,854 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#B#compaction#103 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:58,856 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#A#compaction#102 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:58,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:53:58,893 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 05:53:58,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:58,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:58,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:58,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:58,894 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:58,894 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:58,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:58,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501298918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:58,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501298922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,939 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:58,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:58,941 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:58,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:58,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:58,942 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:58,942 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:58,943 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:58,943 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:58,943 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,943 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:58,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:58,944 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:58,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:58,945 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,945 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:58,945 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:58,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:58,945 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:58,947 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:58,947 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:58,947 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:58,948 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:58,948 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:58,948 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:58,948 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:58,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:58,951 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,951 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:58,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:58,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:58,952 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:58,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ce975b7[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:58,952 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,953 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:58,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:58,954 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:58,954 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,954 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:58,955 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:58,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:58,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:58,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:58,958 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:58,958 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:58,958 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:58,958 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:58,958 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:58,961 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,961 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:58,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,961 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:58,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:58,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,962 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,964 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:58,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:58,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:58,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:58,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:58,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:58,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:58,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:58,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:58,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:58,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:58,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:58,968 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:58,968 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:58,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:58,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:58,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:59,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501299025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:59,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501299025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,068 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:59,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:59,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:59,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:59,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:59,076 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:59,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:59,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:59,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,081 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,082 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:59,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,082 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:59,082 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:59,082 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:59,083 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:59,083 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,083 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:59,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:59,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:59,084 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,084 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:59,084 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:59,084 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:59,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:59,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:59,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:59,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,086 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:59,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60423239[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:59,087 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:59,087 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:59,087 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,087 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:59,089 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,089 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,089 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:59,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:59,094 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,094 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,094 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:59,094 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:59,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:59,099 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:59,100 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:59,100 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:59,100 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:59,100 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:59,106 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,106 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,106 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:59,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:59,107 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,109 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,112 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,112 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:59,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,113 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:59,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:59,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:59,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,114 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:59,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:59,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,114 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:59,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:59,214 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:59,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:59,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:59,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:59,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:59,219 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:59,219 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,219 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:59,219 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:59,220 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,220 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,222 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,222 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,222 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:59,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:59,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:59,225 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:59,225 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:59,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:59,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:59,228 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,228 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:59,228 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:59,228 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:59,228 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:59,230 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,231 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,231 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:59,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77b5d758[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:59,233 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:59,233 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,234 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,234 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:59,234 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:59,234 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:59,234 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,234 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:59,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:59,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501299229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,236 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:59,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501299230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,237 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:59,237 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,237 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,237 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:59,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:59,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:59,241 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:59,241 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:59,241 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:59,241 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:59,241 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:59,244 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,244 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,244 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,245 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,249 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,249 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,249 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,249 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:59,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:59,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:59,249 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,249 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,249 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,249 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,249 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,249 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:59,249 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:59,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:59,249 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:59,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,270 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/e61f1fe06bcd4b299d67313ec1b9b2df as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/e61f1fe06bcd4b299d67313ec1b9b2df 2023-04-03 05:53:59,276 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/B of 6dcb9105330925690aad25e3fde3c83c into e61f1fe06bcd4b299d67313ec1b9b2df(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:59,276 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:59,276 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/B, priority=8, startTime=1680501238824; duration=0sec 2023-04-03 05:53:59,277 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:53:59,277 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-03 05:53:59,279 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/210b844ba6d4498aa6cecf8f3ef0bf5f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/210b844ba6d4498aa6cecf8f3ef0bf5f 2023-04-03 05:53:59,285 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 95930 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-03 05:53:59,285 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/C is initiating minor compaction (all files) 2023-04-03 05:53:59,285 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/C in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,285 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/a2ae1171ff0a4893ad74e9164a8ec622, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/32baea2932d44eb78dcae83769d6734d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/acf059be4a23422db8647e9e87f78309, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/9a3b6b8f405349bdba0e776886ce4ed8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/bcf1b698d9b24c15b7788a5a65cf8239, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/ffec45ff61834421b6273449f9164f05, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/1e882e98ef184ecebe0e89cc29ef4595, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/348d8a0f4a654739bae07b1e6feea0ce] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=93.7 K 2023-04-03 05:53:59,287 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a2ae1171ff0a4893ad74e9164a8ec622, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=372, earliestPutTs=1680501232225 2023-04-03 05:53:59,288 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 32baea2932d44eb78dcae83769d6734d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680501232555 2023-04-03 05:53:59,290 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting acf059be4a23422db8647e9e87f78309, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=407, earliestPutTs=1680501233709 2023-04-03 05:53:59,290 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 9a3b6b8f405349bdba0e776886ce4ed8, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680501234837 2023-04-03 05:53:59,291 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting bcf1b698d9b24c15b7788a5a65cf8239, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=439, earliestPutTs=1680501235174 2023-04-03 05:53:59,291 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ffec45ff61834421b6273449f9164f05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680501235811 2023-04-03 05:53:59,294 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/A of 6dcb9105330925690aad25e3fde3c83c into 210b844ba6d4498aa6cecf8f3ef0bf5f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:59,294 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:59,294 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/A, priority=8, startTime=1680501238824; duration=0sec 2023-04-03 05:53:59,294 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:59,294 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1e882e98ef184ecebe0e89cc29ef4595, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1680501236454 2023-04-03 05:53:59,295 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 348d8a0f4a654739bae07b1e6feea0ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680501237596 2023-04-03 05:53:59,305 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/3a8c849533d447fbb857a7003b5580d7 2023-04-03 05:53:59,322 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#C#compaction#105 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:53:59,347 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/08178a4b373c48d7a3942a53674b0857 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/08178a4b373c48d7a3942a53674b0857 2023-04-03 05:53:59,349 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:59,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:59,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:59,356 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/C of 6dcb9105330925690aad25e3fde3c83c into 08178a4b373c48d7a3942a53674b0857(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:53:59,356 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:59,356 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/C, priority=8, startTime=1680501238824; duration=0sec 2023-04-03 05:53:59,356 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:53:59,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:59,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:59,359 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:59,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:59,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:59,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:59,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,362 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:59,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:59,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:59,363 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,363 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:59,363 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:59,363 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:59,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:59,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:59,365 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,365 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,366 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,366 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:59,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f94a1c4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:59,369 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:59,369 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:59,372 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:59,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:59,374 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:59,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,374 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,374 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:59,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:59,379 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:59,379 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:59,379 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:59,379 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:59,379 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:59,381 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,381 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:59,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,381 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,382 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,384 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,386 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,386 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,386 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,387 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,387 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:59,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:59,387 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:59,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,387 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:59,387 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,387 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,387 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,387 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:59,387 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:59,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:59,487 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:59,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:59,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:59,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:59,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:59,491 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:59,491 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,491 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:59,491 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:59,491 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,491 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,493 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,493 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,494 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:59,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:59,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:59,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:59,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:59,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:59,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:59,495 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,495 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:59,495 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:59,495 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:59,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:59,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:59,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:59,497 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,497 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,498 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,498 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:59,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:59,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d97f61e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:59,498 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:59,499 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,502 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,502 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:59,502 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:59,503 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:59,503 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,503 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:59,506 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,506 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:59,506 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,506 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:59,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:59,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:59,508 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:59,508 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:59,508 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:59,508 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:59,508 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:59,510 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,510 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:59,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,512 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,514 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:59,515 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,515 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,515 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,515 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:59,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:59,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:59,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,515 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:59,515 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:59,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:59,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501299537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,539 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:53:59,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501299537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,615 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:59,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:59,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:59,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:59,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:59,618 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:59,618 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,618 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:59,618 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:59,619 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,619 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,621 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:59,622 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:59,622 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:59,622 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:59,622 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,622 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:59,622 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:59,622 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:59,622 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,622 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:59,622 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:59,622 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,622 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:59,622 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:59,622 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,622 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:59,622 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:59,624 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,624 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,625 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,625 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:59,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a547cd6[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:59,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:59,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,627 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,627 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:59,627 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:59,628 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:59,628 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,628 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:59,629 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,629 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:59,629 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,629 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:53:59,631 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:59,631 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:59,631 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:59,631 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:59,631 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:59,631 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:59,633 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,633 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,633 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:59,634 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,638 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,638 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,638 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,638 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:59,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:59,638 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:59,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,638 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:59,638 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,639 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:59,639 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:59,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,739 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:59,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/c63182bb20a640cbb1ab188a0c5f0bd8 2023-04-03 05:53:59,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:59,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:59,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:59,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:59,743 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:59,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:59,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:59,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,746 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,746 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:59,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:59,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:59,749 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:59,749 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:59,749 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:59,751 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,751 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:59,751 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:53:59,751 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:53:59,751 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:53:59,753 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,753 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:53:59,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@430277d3[Count = 0] remaining members to acquire global barrier 2023-04-03 05:53:59,755 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:53:59,755 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,756 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:53:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:53:59,756 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:53:59,758 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=506 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/c01a50206f1645aab00ac716ae57f674 2023-04-03 05:53:59,759 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:53:59,759 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:59,760 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:53:59,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:53:59,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:53:59,760 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:53:59,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:53:59,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:53:59,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:53:59,760 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:53:59,762 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,762 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,762 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:53:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:53:59,763 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:53:59,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:53:59,764 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/3a8c849533d447fbb857a7003b5580d7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3a8c849533d447fbb857a7003b5580d7 2023-04-03 05:53:59,765 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:53:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:53:59,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,768 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,768 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:53:59,768 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:53:59,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:53:59,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,768 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,768 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,768 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:53:59,768 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:53:59,768 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:53:59,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:53:59,768 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:53:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:53:59,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3a8c849533d447fbb857a7003b5580d7, entries=250, sequenceid=506, filesize=16.7 K 2023-04-03 05:53:59,769 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,769 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:53:59,769 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/c63182bb20a640cbb1ab188a0c5f0bd8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/c63182bb20a640cbb1ab188a0c5f0bd8 2023-04-03 05:53:59,789 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/c63182bb20a640cbb1ab188a0c5f0bd8, entries=150, sequenceid=506, filesize=11.9 K 2023-04-03 05:53:59,790 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/c01a50206f1645aab00ac716ae57f674 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/c01a50206f1645aab00ac716ae57f674 2023-04-03 05:53:59,796 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/c01a50206f1645aab00ac716ae57f674, entries=150, sequenceid=506, filesize=11.9 K 2023-04-03 05:53:59,798 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 6dcb9105330925690aad25e3fde3c83c in 904ms, sequenceid=506, compaction requested=false 2023-04-03 05:53:59,798 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:53:59,868 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:53:59,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:53:59,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:53:59,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:53:59,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:53:59,872 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:53:59,872 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,872 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:53:59,872 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:53:59,872 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,873 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,874 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:53:59,874 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:53:59,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:53:59,874 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:53:59,874 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:53:59,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,875 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:53:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:53:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:53:59,875 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:53:59,875 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:53:59,876 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:53:59,876 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:53:59,876 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:53:59,876 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:53:59,876 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:53:59,876 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:53:59,876 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:53:59,876 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. started... 2023-04-03 05:53:59,877 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 05:53:59,877 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:53:59,877 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:59,877 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:53:59,877 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:53:59,877 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:53:59,877 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:00,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:54:00,042 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. as already flushing 2023-04-03 05:54:00,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:00,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501300070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:00,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:00,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501300071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:00,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:00,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:00,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501300174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:00,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501300174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:00,249 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ec2fae8 to 127.0.0.1:63744 2023-04-03 05:54:00,249 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:00,249 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27685d29 to 127.0.0.1:63744 2023-04-03 05:54:00,249 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:00,250 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5979620a to 127.0.0.1:63744 2023-04-03 05:54:00,250 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:00,253 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00375c9d to 127.0.0.1:63744 2023-04-03 05:54:00,253 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:00,294 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/0712a3678a4d4aacb11949194f52f85f 2023-04-03 05:54:00,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:00,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501300376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:00,377 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:00,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501300377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:00,679 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:00,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:00,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47792 deadline: 1680501300678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:00,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47768 deadline: 1680501300679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:00,706 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/dea233c15d2747838dd63e7353e9ca92 2023-04-03 05:54:01,117 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=521 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/fb9384b59f184e0ebb3dfa9babed632e 2023-04-03 05:54:01,122 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/0712a3678a4d4aacb11949194f52f85f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/0712a3678a4d4aacb11949194f52f85f 2023-04-03 05:54:01,126 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/0712a3678a4d4aacb11949194f52f85f, entries=100, sequenceid=521, filesize=9.5 K 2023-04-03 05:54:01,127 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/dea233c15d2747838dd63e7353e9ca92 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/dea233c15d2747838dd63e7353e9ca92 2023-04-03 05:54:01,131 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/dea233c15d2747838dd63e7353e9ca92, entries=100, sequenceid=521, filesize=9.5 K 2023-04-03 05:54:01,131 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/fb9384b59f184e0ebb3dfa9babed632e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/fb9384b59f184e0ebb3dfa9babed632e 2023-04-03 05:54:01,136 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/fb9384b59f184e0ebb3dfa9babed632e, entries=100, sequenceid=521, filesize=9.5 K 2023-04-03 05:54:01,137 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 6dcb9105330925690aad25e3fde3c83c in 1260ms, sequenceid=521, compaction requested=true 2023-04-03 05:54:01,137 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:54:01,137 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:54:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:01,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:01,139 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,139 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:01,140 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:01,140 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:01,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:01,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29f274f6[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:01,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:01,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:01,147 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:01,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:01,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:01,147 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:01,147 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:01,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:01,147 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:01,148 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:01,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:01,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:01,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:01,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:01,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:01,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:01,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:01,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:01,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:01,150 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:01,150 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:01,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:01,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:01,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:01,151 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:01,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:01,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:01,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:01,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:01,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:01,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:01,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:01,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:01,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:01,153 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:01,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:01,154 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:01,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:01,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:01,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:01,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:01,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:01,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:01,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:01,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:01,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:01,156 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:01,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:01,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:01,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:01,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:01,156 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:01,156 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:01,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:01,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:01,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:01,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:54:01,184 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-03 05:54:01,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:54:01,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:01,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:54:01,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:01,184 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:54:01,184 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:01,185 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7b52aa42 to 127.0.0.1:63744 2023-04-03 05:54:01,185 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:01,186 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7bc6d729 to 127.0.0.1:63744 2023-04-03 05:54:01,186 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:01,256 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:01,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:01,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/3e42ccdc89a942fdaa0c759137a62bb9 2023-04-03 05:54:02,005 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/97735c2bb5c14bb6be07eeb8415f38d5 2023-04-03 05:54:02,417 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/5dee108ae6124699934d3922980236f2 2023-04-03 05:54:02,421 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/3e42ccdc89a942fdaa0c759137a62bb9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3e42ccdc89a942fdaa0c759137a62bb9 2023-04-03 05:54:02,425 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3e42ccdc89a942fdaa0c759137a62bb9, entries=150, sequenceid=542, filesize=11.9 K 2023-04-03 05:54:02,426 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/97735c2bb5c14bb6be07eeb8415f38d5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/97735c2bb5c14bb6be07eeb8415f38d5 2023-04-03 05:54:02,430 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/97735c2bb5c14bb6be07eeb8415f38d5, entries=150, sequenceid=542, filesize=11.9 K 2023-04-03 05:54:02,431 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/5dee108ae6124699934d3922980236f2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/5dee108ae6124699934d3922980236f2 2023-04-03 05:54:02,434 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/5dee108ae6124699934d3922980236f2, entries=150, sequenceid=542, filesize=11.9 K 2023-04-03 05:54:02,435 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=6.71 KB/6870 for 6dcb9105330925690aad25e3fde3c83c in 1251ms, sequenceid=542, compaction requested=true 2023-04-03 05:54:02,435 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:54:02,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:02,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:02,435 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:02,435 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:02,435 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:02,437 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52338 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:02,437 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47458 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:02,437 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/A is initiating minor compaction (all files) 2023-04-03 05:54:02,437 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/B is initiating minor compaction (all files) 2023-04-03 05:54:02,437 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/A in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:54:02,437 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/B in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:54:02,437 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/210b844ba6d4498aa6cecf8f3ef0bf5f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3a8c849533d447fbb857a7003b5580d7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/0712a3678a4d4aacb11949194f52f85f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3e42ccdc89a942fdaa0c759137a62bb9] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=51.1 K 2023-04-03 05:54:02,437 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/e61f1fe06bcd4b299d67313ec1b9b2df, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/c63182bb20a640cbb1ab188a0c5f0bd8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/dea233c15d2747838dd63e7353e9ca92, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/97735c2bb5c14bb6be07eeb8415f38d5] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=46.3 K 2023-04-03 05:54:02,437 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 210b844ba6d4498aa6cecf8f3ef0bf5f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680501237596 2023-04-03 05:54:02,437 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e61f1fe06bcd4b299d67313ec1b9b2df, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680501237596 2023-04-03 05:54:02,438 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 3a8c849533d447fbb857a7003b5580d7, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680501238755 2023-04-03 05:54:02,438 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting c63182bb20a640cbb1ab188a0c5f0bd8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680501238777 2023-04-03 05:54:02,438 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 0712a3678a4d4aacb11949194f52f85f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680501238917 2023-04-03 05:54:02,438 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting dea233c15d2747838dd63e7353e9ca92, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680501238917 2023-04-03 05:54:02,439 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 3e42ccdc89a942fdaa0c759137a62bb9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1680501240066 2023-04-03 05:54:02,439 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 97735c2bb5c14bb6be07eeb8415f38d5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1680501240066 2023-04-03 05:54:02,454 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#B#compaction#115 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:02,454 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#A#compaction#114 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:02,870 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/7cc83c17d3274da2bdfef3b8713aade1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/7cc83c17d3274da2bdfef3b8713aade1 2023-04-03 05:54:02,870 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/13661f2e093147ceb93c273271c073fe as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/13661f2e093147ceb93c273271c073fe 2023-04-03 05:54:02,876 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/A of 6dcb9105330925690aad25e3fde3c83c into 13661f2e093147ceb93c273271c073fe(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:02,876 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/B of 6dcb9105330925690aad25e3fde3c83c into 7cc83c17d3274da2bdfef3b8713aade1(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:02,876 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:54:02,876 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:54:02,877 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/A, priority=12, startTime=1680501242435; duration=0sec 2023-04-03 05:54:02,877 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:02,877 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:02,877 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/B, priority=12, startTime=1680501242435; duration=0sec 2023-04-03 05:54:02,877 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:02,878 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47458 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:02,879 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 6dcb9105330925690aad25e3fde3c83c/C is initiating minor compaction (all files) 2023-04-03 05:54:02,879 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 6dcb9105330925690aad25e3fde3c83c/C in TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:54:02,879 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/08178a4b373c48d7a3942a53674b0857, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/c01a50206f1645aab00ac716ae57f674, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/fb9384b59f184e0ebb3dfa9babed632e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/5dee108ae6124699934d3922980236f2] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp, totalSize=46.3 K 2023-04-03 05:54:02,879 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 08178a4b373c48d7a3942a53674b0857, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=486, earliestPutTs=1680501237596 2023-04-03 05:54:02,880 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting c01a50206f1645aab00ac716ae57f674, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=506, earliestPutTs=1680501238777 2023-04-03 05:54:02,880 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting fb9384b59f184e0ebb3dfa9babed632e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=521, earliestPutTs=1680501238917 2023-04-03 05:54:02,881 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 5dee108ae6124699934d3922980236f2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=542, earliestPutTs=1680501240066 2023-04-03 05:54:02,890 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 6dcb9105330925690aad25e3fde3c83c#C#compaction#116 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:02,901 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/ac053cd5ef954747aeb478fba688787c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/ac053cd5ef954747aeb478fba688787c 2023-04-03 05:54:02,906 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 6dcb9105330925690aad25e3fde3c83c/C of 6dcb9105330925690aad25e3fde3c83c into ac053cd5ef954747aeb478fba688787c(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:02,906 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:54:02,906 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c., storeName=6dcb9105330925690aad25e3fde3c83c/C, priority=12, startTime=1680501242435; duration=0sec 2023-04-03 05:54:02,906 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:06,638 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1645b6b0 to 127.0.0.1:63744 2023-04-03 05:54:06,638 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:06,673 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1710407d to 127.0.0.1:63744 2023-04-03 05:54:06,673 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:06,684 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x72ea4a65 to 127.0.0.1:63744 2023-04-03 05:54:06,684 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:06,684 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 41 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 140 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 49 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 151 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 4522 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 4480 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 1967 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 5897 rows 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 1974 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 5917 rows 2023-04-03 05:54:06,685 INFO [Listener at localhost/37275] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 05:54:06,685 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5d4f988c to 127.0.0.1:63744 2023-04-03 05:54:06,685 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:06,690 INFO [Listener at localhost/37275] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 05:54:06,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-03 05:54:06,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:06,704 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501246703"}]},"ts":"1680501246703"} 2023-04-03 05:54:06,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 05:54:06,705 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 05:54:06,706 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 05:54:06,708 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dcb9105330925690aad25e3fde3c83c, UNASSIGN}] 2023-04-03 05:54:06,709 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dcb9105330925690aad25e3fde3c83c, UNASSIGN 2023-04-03 05:54:06,710 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6dcb9105330925690aad25e3fde3c83c, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:06,710 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501246710"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501246710"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501246710"}]},"ts":"1680501246710"} 2023-04-03 05:54:06,711 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:54:06,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 05:54:06,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:54:06,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6dcb9105330925690aad25e3fde3c83c, disabling compactions & flushes 2023-04-03 05:54:06,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:54:06,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:54:06,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. after waiting 0 ms 2023-04-03 05:54:06,869 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:54:06,869 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6dcb9105330925690aad25e3fde3c83c 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 05:54:06,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=A 2023-04-03 05:54:06,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:06,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=B 2023-04-03 05:54:06,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:06,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 6dcb9105330925690aad25e3fde3c83c, store=C 2023-04-03 05:54:06,870 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:07,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 05:54:07,282 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/8428fd8e7a4546eba8b91155c32d76cb 2023-04-03 05:54:07,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 05:54:07,694 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/6613422dc94c45aeaa1e258bc82796c4 2023-04-03 05:54:07,705 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=552 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/7cf82f788c8a490e9874fdf00ab1c0e8 2023-04-03 05:54:07,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/A/8428fd8e7a4546eba8b91155c32d76cb as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/8428fd8e7a4546eba8b91155c32d76cb 2023-04-03 05:54:07,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/8428fd8e7a4546eba8b91155c32d76cb, entries=150, sequenceid=552, filesize=11.9 K 2023-04-03 05:54:07,716 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/B/6613422dc94c45aeaa1e258bc82796c4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/6613422dc94c45aeaa1e258bc82796c4 2023-04-03 05:54:07,721 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/6613422dc94c45aeaa1e258bc82796c4, entries=150, sequenceid=552, filesize=11.9 K 2023-04-03 05:54:07,721 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/.tmp/C/7cf82f788c8a490e9874fdf00ab1c0e8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/7cf82f788c8a490e9874fdf00ab1c0e8 2023-04-03 05:54:07,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/7cf82f788c8a490e9874fdf00ab1c0e8, entries=150, sequenceid=552, filesize=11.9 K 2023-04-03 05:54:07,727 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 6dcb9105330925690aad25e3fde3c83c in 858ms, sequenceid=552, compaction requested=false 2023-04-03 05:54:07,749 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/fe1efb84860b4668915a5cc5dea3bf98, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/8375e37082064e99b3c4306e8eb29de9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3da0d3d1b7ae41de8f021dcd47fd2ba7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/38d0d94e57d847329f9132381d2ca4b2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/83a6d6fba6ff438085aa4c3ce00791b2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/813182bb7b594b7cae410dd61364a27a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/73c748a164c54c97a9f4f4b27ab82014, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/487df9cd051947bbacf9acb50f35c5e4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/ba09a0e9b53a4e59a3356a35533dfd58, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/f701e37734c74514af2625013b3b4342, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/deca8407e2dc42868cff033bce3634a7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5eb94091ec7248e382483d1c9442f01b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/a32b57efdd14440a877f6e43394d0d7a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/acff5124d1f74aa8b442fb46260a94c2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/e3cd9d3a6cbb4ed19b77cc43a6298f48, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/52c1926fd7614c15ac66745b6e735d12, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/157bf3f585454ad4adfea883597907f9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/a0c11360fb434b4081d96fd0d6d872bc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/7f2643d5f19a40dcac83fa7449bf0a55, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5415e8c4796b487aa50a027ed6d96988, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/209641050f1246769a9c28ab24d8f8b2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/c922b16b246046f4ba2bd56dc91d32f3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/c4c5074970cb49f0b866b9ea41d5aae4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/7a905929bf0144f7bc98513a97578ab1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/92ffe7c15a9b4034a4aaf5584cc10be7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/eab0428996754a57a9dd569513613466, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/bd7b1e836107469c8b364c0891e8318d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/439f27c4833b4c528faaf3652e7c42ab, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/49913aa6f53b47d9b9df055a4ce481d4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/360ffaed2e2249958eb6510568249c31, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/9d3e1a84f3134341a82560078395a8dd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/239f4ef397b645fa9fe3d26a4c0b646a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5518a9fb1dac4859abc08343dc8e8b19, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/52debfab7b974bff8f648a0bd1861980, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/210b844ba6d4498aa6cecf8f3ef0bf5f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3a8c849533d447fbb857a7003b5580d7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/0712a3678a4d4aacb11949194f52f85f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3e42ccdc89a942fdaa0c759137a62bb9] to archive 2023-04-03 05:54:07,750 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:54:07,756 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/fe1efb84860b4668915a5cc5dea3bf98 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/fe1efb84860b4668915a5cc5dea3bf98 2023-04-03 05:54:07,757 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/8375e37082064e99b3c4306e8eb29de9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/8375e37082064e99b3c4306e8eb29de9 2023-04-03 05:54:07,758 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3da0d3d1b7ae41de8f021dcd47fd2ba7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3da0d3d1b7ae41de8f021dcd47fd2ba7 2023-04-03 05:54:07,760 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/38d0d94e57d847329f9132381d2ca4b2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/38d0d94e57d847329f9132381d2ca4b2 2023-04-03 05:54:07,761 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/83a6d6fba6ff438085aa4c3ce00791b2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/83a6d6fba6ff438085aa4c3ce00791b2 2023-04-03 05:54:07,763 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/813182bb7b594b7cae410dd61364a27a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/813182bb7b594b7cae410dd61364a27a 2023-04-03 05:54:07,764 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/73c748a164c54c97a9f4f4b27ab82014 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/73c748a164c54c97a9f4f4b27ab82014 2023-04-03 05:54:07,765 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/487df9cd051947bbacf9acb50f35c5e4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/487df9cd051947bbacf9acb50f35c5e4 2023-04-03 05:54:07,766 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/ba09a0e9b53a4e59a3356a35533dfd58 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/ba09a0e9b53a4e59a3356a35533dfd58 2023-04-03 05:54:07,767 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/f701e37734c74514af2625013b3b4342 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/f701e37734c74514af2625013b3b4342 2023-04-03 05:54:07,769 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/deca8407e2dc42868cff033bce3634a7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/deca8407e2dc42868cff033bce3634a7 2023-04-03 05:54:07,770 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5eb94091ec7248e382483d1c9442f01b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5eb94091ec7248e382483d1c9442f01b 2023-04-03 05:54:07,771 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/a32b57efdd14440a877f6e43394d0d7a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/a32b57efdd14440a877f6e43394d0d7a 2023-04-03 05:54:07,773 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/acff5124d1f74aa8b442fb46260a94c2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/acff5124d1f74aa8b442fb46260a94c2 2023-04-03 05:54:07,774 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/e3cd9d3a6cbb4ed19b77cc43a6298f48 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/e3cd9d3a6cbb4ed19b77cc43a6298f48 2023-04-03 05:54:07,776 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/52c1926fd7614c15ac66745b6e735d12 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/52c1926fd7614c15ac66745b6e735d12 2023-04-03 05:54:07,779 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/157bf3f585454ad4adfea883597907f9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/157bf3f585454ad4adfea883597907f9 2023-04-03 05:54:07,780 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/a0c11360fb434b4081d96fd0d6d872bc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/a0c11360fb434b4081d96fd0d6d872bc 2023-04-03 05:54:07,781 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/7f2643d5f19a40dcac83fa7449bf0a55 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/7f2643d5f19a40dcac83fa7449bf0a55 2023-04-03 05:54:07,782 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5415e8c4796b487aa50a027ed6d96988 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5415e8c4796b487aa50a027ed6d96988 2023-04-03 05:54:07,784 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/209641050f1246769a9c28ab24d8f8b2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/209641050f1246769a9c28ab24d8f8b2 2023-04-03 05:54:07,785 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/c922b16b246046f4ba2bd56dc91d32f3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/c922b16b246046f4ba2bd56dc91d32f3 2023-04-03 05:54:07,786 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/c4c5074970cb49f0b866b9ea41d5aae4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/c4c5074970cb49f0b866b9ea41d5aae4 2023-04-03 05:54:07,787 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/7a905929bf0144f7bc98513a97578ab1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/7a905929bf0144f7bc98513a97578ab1 2023-04-03 05:54:07,788 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/92ffe7c15a9b4034a4aaf5584cc10be7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/92ffe7c15a9b4034a4aaf5584cc10be7 2023-04-03 05:54:07,789 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/eab0428996754a57a9dd569513613466 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/eab0428996754a57a9dd569513613466 2023-04-03 05:54:07,790 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/bd7b1e836107469c8b364c0891e8318d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/bd7b1e836107469c8b364c0891e8318d 2023-04-03 05:54:07,791 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/439f27c4833b4c528faaf3652e7c42ab to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/439f27c4833b4c528faaf3652e7c42ab 2023-04-03 05:54:07,792 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/49913aa6f53b47d9b9df055a4ce481d4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/49913aa6f53b47d9b9df055a4ce481d4 2023-04-03 05:54:07,793 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/360ffaed2e2249958eb6510568249c31 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/360ffaed2e2249958eb6510568249c31 2023-04-03 05:54:07,794 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/9d3e1a84f3134341a82560078395a8dd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/9d3e1a84f3134341a82560078395a8dd 2023-04-03 05:54:07,795 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/239f4ef397b645fa9fe3d26a4c0b646a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/239f4ef397b645fa9fe3d26a4c0b646a 2023-04-03 05:54:07,796 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5518a9fb1dac4859abc08343dc8e8b19 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/5518a9fb1dac4859abc08343dc8e8b19 2023-04-03 05:54:07,797 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/52debfab7b974bff8f648a0bd1861980 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/52debfab7b974bff8f648a0bd1861980 2023-04-03 05:54:07,798 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/210b844ba6d4498aa6cecf8f3ef0bf5f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/210b844ba6d4498aa6cecf8f3ef0bf5f 2023-04-03 05:54:07,799 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3a8c849533d447fbb857a7003b5580d7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3a8c849533d447fbb857a7003b5580d7 2023-04-03 05:54:07,800 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/0712a3678a4d4aacb11949194f52f85f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/0712a3678a4d4aacb11949194f52f85f 2023-04-03 05:54:07,802 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3e42ccdc89a942fdaa0c759137a62bb9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/3e42ccdc89a942fdaa0c759137a62bb9 2023-04-03 05:54:07,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 05:54:07,834 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/61f713fcc4034c8cb2b673c12cf73653, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/3466e3a4a518406795088acc0c6e5ae8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/9b9e8315b9034159afd47780b33448d2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/6c874ce13a5b41839861d3391ee60ac5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/de616e914eac41e796ef1f99d8eabb26, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/9fb824fbe5bd4c4db84b8b0cea60213a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/3f25d7f8f1724db380977dbb1553ec9e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/0e7de16c3f08498bbc5f4f58b656bbaa, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/4f6c156f926b4563b5f5f6d5352ad01a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/e3a5778da2a942cb84a6267f024efdb0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/8d41b636e74242448914deb63a356f9b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b87b5c9f6ccc4b09b38ec00400cade63, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/c64f73c360a64b2597a9f46fdf9bbc06, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/ab5db2c70b2b4bd7a9ec7615834100b4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/62bbefade5e24051867a4875a93a3f60, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/969eec029adb40e09009550232aedb59, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b2763e9732174c86b02d5dc4251370e1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b42ab5f7d19f49b391ff22a0b0288cd0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/0f18767820084fb5ab9d2d5040815556, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/90ef1f21e18446738f280dfd528ab766, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/5c3a2c238eb54836894badc279e733dc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/04b8fd4ab0394120bb222b06ea0bcc47, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/af78dd718a654e8faff123e2f30b5a43, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/1fe7ca60304a49739383b8b9e8b5f489, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/06d0204fe069437da3139990ab367081, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/02d9994e37fd44ba83e6a77379321bb1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b5704d997ef04d2b93f103cf50f36080, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/70bbe05888524af28cbec2dbaa06c2bb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/a70d734426414591ae45673da16f0fe4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/14b69eb9f122408680d9e61d926a5bd2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/fee978b8176443d9aff625d351fd9819, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/96e4f3164e964a3b8f8447188b3d64fa, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/8dee24a8c30e4a3dae2dc4ba91ff9110, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/e61f1fe06bcd4b299d67313ec1b9b2df, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/bb0681573e1241c6aafec1e115d176e8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/c63182bb20a640cbb1ab188a0c5f0bd8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/dea233c15d2747838dd63e7353e9ca92, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/97735c2bb5c14bb6be07eeb8415f38d5] to archive 2023-04-03 05:54:07,835 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:54:07,837 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/61f713fcc4034c8cb2b673c12cf73653 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/61f713fcc4034c8cb2b673c12cf73653 2023-04-03 05:54:07,839 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/3466e3a4a518406795088acc0c6e5ae8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/3466e3a4a518406795088acc0c6e5ae8 2023-04-03 05:54:07,840 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/9b9e8315b9034159afd47780b33448d2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/9b9e8315b9034159afd47780b33448d2 2023-04-03 05:54:07,842 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/6c874ce13a5b41839861d3391ee60ac5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/6c874ce13a5b41839861d3391ee60ac5 2023-04-03 05:54:07,844 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/de616e914eac41e796ef1f99d8eabb26 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/de616e914eac41e796ef1f99d8eabb26 2023-04-03 05:54:07,845 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/9fb824fbe5bd4c4db84b8b0cea60213a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/9fb824fbe5bd4c4db84b8b0cea60213a 2023-04-03 05:54:07,846 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/3f25d7f8f1724db380977dbb1553ec9e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/3f25d7f8f1724db380977dbb1553ec9e 2023-04-03 05:54:07,848 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/0e7de16c3f08498bbc5f4f58b656bbaa to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/0e7de16c3f08498bbc5f4f58b656bbaa 2023-04-03 05:54:07,849 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/4f6c156f926b4563b5f5f6d5352ad01a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/4f6c156f926b4563b5f5f6d5352ad01a 2023-04-03 05:54:07,850 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/e3a5778da2a942cb84a6267f024efdb0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/e3a5778da2a942cb84a6267f024efdb0 2023-04-03 05:54:07,851 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/8d41b636e74242448914deb63a356f9b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/8d41b636e74242448914deb63a356f9b 2023-04-03 05:54:07,853 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b87b5c9f6ccc4b09b38ec00400cade63 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b87b5c9f6ccc4b09b38ec00400cade63 2023-04-03 05:54:07,855 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/c64f73c360a64b2597a9f46fdf9bbc06 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/c64f73c360a64b2597a9f46fdf9bbc06 2023-04-03 05:54:07,856 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/ab5db2c70b2b4bd7a9ec7615834100b4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/ab5db2c70b2b4bd7a9ec7615834100b4 2023-04-03 05:54:07,858 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/62bbefade5e24051867a4875a93a3f60 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/62bbefade5e24051867a4875a93a3f60 2023-04-03 05:54:07,859 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/969eec029adb40e09009550232aedb59 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/969eec029adb40e09009550232aedb59 2023-04-03 05:54:07,860 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b2763e9732174c86b02d5dc4251370e1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b2763e9732174c86b02d5dc4251370e1 2023-04-03 05:54:07,861 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b42ab5f7d19f49b391ff22a0b0288cd0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b42ab5f7d19f49b391ff22a0b0288cd0 2023-04-03 05:54:07,863 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/0f18767820084fb5ab9d2d5040815556 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/0f18767820084fb5ab9d2d5040815556 2023-04-03 05:54:07,864 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/90ef1f21e18446738f280dfd528ab766 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/90ef1f21e18446738f280dfd528ab766 2023-04-03 05:54:07,866 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/5c3a2c238eb54836894badc279e733dc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/5c3a2c238eb54836894badc279e733dc 2023-04-03 05:54:07,867 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/04b8fd4ab0394120bb222b06ea0bcc47 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/04b8fd4ab0394120bb222b06ea0bcc47 2023-04-03 05:54:07,869 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/af78dd718a654e8faff123e2f30b5a43 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/af78dd718a654e8faff123e2f30b5a43 2023-04-03 05:54:07,870 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/1fe7ca60304a49739383b8b9e8b5f489 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/1fe7ca60304a49739383b8b9e8b5f489 2023-04-03 05:54:07,871 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/06d0204fe069437da3139990ab367081 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/06d0204fe069437da3139990ab367081 2023-04-03 05:54:07,873 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/02d9994e37fd44ba83e6a77379321bb1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/02d9994e37fd44ba83e6a77379321bb1 2023-04-03 05:54:07,874 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b5704d997ef04d2b93f103cf50f36080 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/b5704d997ef04d2b93f103cf50f36080 2023-04-03 05:54:07,875 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/70bbe05888524af28cbec2dbaa06c2bb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/70bbe05888524af28cbec2dbaa06c2bb 2023-04-03 05:54:07,876 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/a70d734426414591ae45673da16f0fe4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/a70d734426414591ae45673da16f0fe4 2023-04-03 05:54:07,877 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/14b69eb9f122408680d9e61d926a5bd2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/14b69eb9f122408680d9e61d926a5bd2 2023-04-03 05:54:07,879 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/fee978b8176443d9aff625d351fd9819 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/fee978b8176443d9aff625d351fd9819 2023-04-03 05:54:07,880 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/96e4f3164e964a3b8f8447188b3d64fa to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/96e4f3164e964a3b8f8447188b3d64fa 2023-04-03 05:54:07,881 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/8dee24a8c30e4a3dae2dc4ba91ff9110 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/8dee24a8c30e4a3dae2dc4ba91ff9110 2023-04-03 05:54:07,882 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/e61f1fe06bcd4b299d67313ec1b9b2df to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/e61f1fe06bcd4b299d67313ec1b9b2df 2023-04-03 05:54:07,883 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/bb0681573e1241c6aafec1e115d176e8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/bb0681573e1241c6aafec1e115d176e8 2023-04-03 05:54:07,884 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/c63182bb20a640cbb1ab188a0c5f0bd8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/c63182bb20a640cbb1ab188a0c5f0bd8 2023-04-03 05:54:07,885 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/dea233c15d2747838dd63e7353e9ca92 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/dea233c15d2747838dd63e7353e9ca92 2023-04-03 05:54:07,886 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/97735c2bb5c14bb6be07eeb8415f38d5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/97735c2bb5c14bb6be07eeb8415f38d5 2023-04-03 05:54:07,890 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/e82d73f169c949e79a62185dc86307f3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/be91a306e15148e6a60e141d5b89ea22, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/598eb85158cb4d349bc5142321274d77, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/a6ff5256373b4c528ca840ed2f6d5400, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/41cf6db376554b4590d4cec0ccfceae2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/39831166e8684a838b38a9c289feeadf, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/83c110a8ec384d93aa06b6eb4a4cc927, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/c71830570f134793a1a863b2f056bd4a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/e3789644da5b4eda914c7e822743f3e0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/b155d984a46a4fa0bd87ecf630564c92, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/2db6b11e318c453ab7a109bb71383c82, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/6bebe12f735043fdb035e3cdcc34e1e5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/5a792de166e3409c8c2b7d2c837437df, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/bc9d3301927843bda621ae8689e2cad3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/2e6c7cf1f31d4cf691b99aa0d139a4c5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/38940e4819b9411e8f99ff36488b1170, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/f19487f5c1064ed9b7cd22ee43c987e5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/8f9ae97794524f6e9c58729a3a3ccc14, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/986964e83bbf4a0b9afa8f9b456c5133, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/068deae8a77c43baafc26ced11964da7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/38330f764b414bd590588552c0f0bc0d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/0810c828ec6046429283ed59d22539b3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/dcad422310cc460ebec7c61897bf7a1a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/d04718ce609248529ddb28c13108757a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/3a59e648c4da411d9bb8414179ae1b85, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/a2ae1171ff0a4893ad74e9164a8ec622, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/b7b0644242974d0f8b07d0d4cba467fe, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/32baea2932d44eb78dcae83769d6734d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/acf059be4a23422db8647e9e87f78309, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/9a3b6b8f405349bdba0e776886ce4ed8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/bcf1b698d9b24c15b7788a5a65cf8239, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/ffec45ff61834421b6273449f9164f05, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/1e882e98ef184ecebe0e89cc29ef4595, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/08178a4b373c48d7a3942a53674b0857, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/348d8a0f4a654739bae07b1e6feea0ce, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/c01a50206f1645aab00ac716ae57f674, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/fb9384b59f184e0ebb3dfa9babed632e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/5dee108ae6124699934d3922980236f2] to archive 2023-04-03 05:54:07,891 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:54:07,892 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/e82d73f169c949e79a62185dc86307f3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/e82d73f169c949e79a62185dc86307f3 2023-04-03 05:54:07,894 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/be91a306e15148e6a60e141d5b89ea22 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/be91a306e15148e6a60e141d5b89ea22 2023-04-03 05:54:07,895 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/598eb85158cb4d349bc5142321274d77 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/598eb85158cb4d349bc5142321274d77 2023-04-03 05:54:07,896 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/a6ff5256373b4c528ca840ed2f6d5400 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/a6ff5256373b4c528ca840ed2f6d5400 2023-04-03 05:54:07,897 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/41cf6db376554b4590d4cec0ccfceae2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/41cf6db376554b4590d4cec0ccfceae2 2023-04-03 05:54:07,898 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/39831166e8684a838b38a9c289feeadf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/39831166e8684a838b38a9c289feeadf 2023-04-03 05:54:07,899 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/83c110a8ec384d93aa06b6eb4a4cc927 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/83c110a8ec384d93aa06b6eb4a4cc927 2023-04-03 05:54:07,900 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/c71830570f134793a1a863b2f056bd4a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/c71830570f134793a1a863b2f056bd4a 2023-04-03 05:54:07,901 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/e3789644da5b4eda914c7e822743f3e0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/e3789644da5b4eda914c7e822743f3e0 2023-04-03 05:54:07,902 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/b155d984a46a4fa0bd87ecf630564c92 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/b155d984a46a4fa0bd87ecf630564c92 2023-04-03 05:54:07,903 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/2db6b11e318c453ab7a109bb71383c82 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/2db6b11e318c453ab7a109bb71383c82 2023-04-03 05:54:07,904 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/6bebe12f735043fdb035e3cdcc34e1e5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/6bebe12f735043fdb035e3cdcc34e1e5 2023-04-03 05:54:07,905 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/5a792de166e3409c8c2b7d2c837437df to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/5a792de166e3409c8c2b7d2c837437df 2023-04-03 05:54:07,907 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/bc9d3301927843bda621ae8689e2cad3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/bc9d3301927843bda621ae8689e2cad3 2023-04-03 05:54:07,908 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/2e6c7cf1f31d4cf691b99aa0d139a4c5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/2e6c7cf1f31d4cf691b99aa0d139a4c5 2023-04-03 05:54:07,909 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/38940e4819b9411e8f99ff36488b1170 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/38940e4819b9411e8f99ff36488b1170 2023-04-03 05:54:07,910 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/f19487f5c1064ed9b7cd22ee43c987e5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/f19487f5c1064ed9b7cd22ee43c987e5 2023-04-03 05:54:07,910 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/8f9ae97794524f6e9c58729a3a3ccc14 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/8f9ae97794524f6e9c58729a3a3ccc14 2023-04-03 05:54:07,911 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/986964e83bbf4a0b9afa8f9b456c5133 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/986964e83bbf4a0b9afa8f9b456c5133 2023-04-03 05:54:07,912 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/068deae8a77c43baafc26ced11964da7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/068deae8a77c43baafc26ced11964da7 2023-04-03 05:54:07,913 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/38330f764b414bd590588552c0f0bc0d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/38330f764b414bd590588552c0f0bc0d 2023-04-03 05:54:07,914 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/0810c828ec6046429283ed59d22539b3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/0810c828ec6046429283ed59d22539b3 2023-04-03 05:54:07,915 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/dcad422310cc460ebec7c61897bf7a1a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/dcad422310cc460ebec7c61897bf7a1a 2023-04-03 05:54:07,916 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/d04718ce609248529ddb28c13108757a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/d04718ce609248529ddb28c13108757a 2023-04-03 05:54:07,917 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/3a59e648c4da411d9bb8414179ae1b85 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/3a59e648c4da411d9bb8414179ae1b85 2023-04-03 05:54:07,918 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/a2ae1171ff0a4893ad74e9164a8ec622 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/a2ae1171ff0a4893ad74e9164a8ec622 2023-04-03 05:54:07,919 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/b7b0644242974d0f8b07d0d4cba467fe to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/b7b0644242974d0f8b07d0d4cba467fe 2023-04-03 05:54:07,920 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/32baea2932d44eb78dcae83769d6734d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/32baea2932d44eb78dcae83769d6734d 2023-04-03 05:54:07,922 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/acf059be4a23422db8647e9e87f78309 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/acf059be4a23422db8647e9e87f78309 2023-04-03 05:54:07,922 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/9a3b6b8f405349bdba0e776886ce4ed8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/9a3b6b8f405349bdba0e776886ce4ed8 2023-04-03 05:54:07,923 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/bcf1b698d9b24c15b7788a5a65cf8239 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/bcf1b698d9b24c15b7788a5a65cf8239 2023-04-03 05:54:07,924 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/ffec45ff61834421b6273449f9164f05 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/ffec45ff61834421b6273449f9164f05 2023-04-03 05:54:07,925 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/1e882e98ef184ecebe0e89cc29ef4595 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/1e882e98ef184ecebe0e89cc29ef4595 2023-04-03 05:54:07,926 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/08178a4b373c48d7a3942a53674b0857 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/08178a4b373c48d7a3942a53674b0857 2023-04-03 05:54:07,927 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/348d8a0f4a654739bae07b1e6feea0ce to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/348d8a0f4a654739bae07b1e6feea0ce 2023-04-03 05:54:07,928 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/c01a50206f1645aab00ac716ae57f674 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/c01a50206f1645aab00ac716ae57f674 2023-04-03 05:54:07,929 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/fb9384b59f184e0ebb3dfa9babed632e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/fb9384b59f184e0ebb3dfa9babed632e 2023-04-03 05:54:07,931 DEBUG [StoreCloser-TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/5dee108ae6124699934d3922980236f2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/5dee108ae6124699934d3922980236f2 2023-04-03 05:54:07,935 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/recovered.edits/555.seqid, newMaxSeqId=555, maxSeqId=1 2023-04-03 05:54:07,936 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c. 2023-04-03 05:54:07,936 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6dcb9105330925690aad25e3fde3c83c: 2023-04-03 05:54:07,938 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:54:07,938 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=6dcb9105330925690aad25e3fde3c83c, regionState=CLOSED 2023-04-03 05:54:07,939 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501247938"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501247938"}]},"ts":"1680501247938"} 2023-04-03 05:54:07,941 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-03 05:54:07,941 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 6dcb9105330925690aad25e3fde3c83c, server=jenkins-hbase4.apache.org,37397,1680501214665 in 1.2290 sec 2023-04-03 05:54:07,943 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-03 05:54:07,943 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=6dcb9105330925690aad25e3fde3c83c, UNASSIGN in 1.2330 sec 2023-04-03 05:54:07,944 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501247944"}]},"ts":"1680501247944"} 2023-04-03 05:54:07,945 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 05:54:07,946 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 05:54:07,948 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.2510 sec 2023-04-03 05:54:08,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-03 05:54:08,809 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-03 05:54:08,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-03 05:54:08,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:08,821 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:08,823 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:08,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-03 05:54:08,827 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:54:08,830 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/recovered.edits] 2023-04-03 05:54:08,834 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/13661f2e093147ceb93c273271c073fe to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/13661f2e093147ceb93c273271c073fe 2023-04-03 05:54:08,835 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/8428fd8e7a4546eba8b91155c32d76cb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/A/8428fd8e7a4546eba8b91155c32d76cb 2023-04-03 05:54:08,837 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/6613422dc94c45aeaa1e258bc82796c4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/6613422dc94c45aeaa1e258bc82796c4 2023-04-03 05:54:08,838 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/7cc83c17d3274da2bdfef3b8713aade1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/B/7cc83c17d3274da2bdfef3b8713aade1 2023-04-03 05:54:08,841 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/7cf82f788c8a490e9874fdf00ab1c0e8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/7cf82f788c8a490e9874fdf00ab1c0e8 2023-04-03 05:54:08,842 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/ac053cd5ef954747aeb478fba688787c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/C/ac053cd5ef954747aeb478fba688787c 2023-04-03 05:54:08,846 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/recovered.edits/555.seqid to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c/recovered.edits/555.seqid 2023-04-03 05:54:08,847 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/6dcb9105330925690aad25e3fde3c83c 2023-04-03 05:54:08,847 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 05:54:08,849 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:08,870 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 05:54:08,913 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 05:54:08,915 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:08,915 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 05:54:08,915 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680501248915"}]},"ts":"9223372036854775807"} 2023-04-03 05:54:08,917 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 05:54:08,918 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 6dcb9105330925690aad25e3fde3c83c, NAME => 'TestAcidGuarantees,,1680501219000.6dcb9105330925690aad25e3fde3c83c.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 05:54:08,918 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 05:54:08,918 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680501248918"}]},"ts":"9223372036854775807"} 2023-04-03 05:54:08,920 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 05:54:08,922 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:08,923 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 107 msec 2023-04-03 05:54:08,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-03 05:54:08,925 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-03 05:54:08,944 INFO [Listener at localhost/37275] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=230 (was 206) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x55d29e8f-shared-pool-20 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x55d29e8f-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_1830999008_15 at /127.0.0.1:48596 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x55d29e8f-shared-pool-19 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x55d29e8f-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:37397-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=557 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=359 (was 323) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3866 (was 4461) 2023-04-03 05:54:08,958 INFO [Listener at localhost/37275] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=230, OpenFileDescriptor=557, MaxFileDescriptor=60000, SystemLoadAverage=359, ProcessCount=169, AvailableMemoryMB=3866 2023-04-03 05:54:08,960 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 05:54:08,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 05:54:08,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:08,964 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 05:54:08,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-03 05:54:08,965 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 05:54:08,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 05:54:08,966 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:08,967 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a empty. 2023-04-03 05:54:08,967 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:08,967 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 05:54:09,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 05:54:09,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 05:54:09,380 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 05:54:09,382 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 574b09e98d501def4bb65b12ef4de07a, NAME => 'TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp 2023-04-03 05:54:09,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 05:54:09,789 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:54:09,789 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 574b09e98d501def4bb65b12ef4de07a, disabling compactions & flushes 2023-04-03 05:54:09,789 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:09,789 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:09,789 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. after waiting 0 ms 2023-04-03 05:54:09,789 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:09,789 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:09,789 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:09,791 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 05:54:09,792 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501249792"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501249792"}]},"ts":"1680501249792"} 2023-04-03 05:54:09,794 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 05:54:09,794 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 05:54:09,795 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501249795"}]},"ts":"1680501249795"} 2023-04-03 05:54:09,796 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 05:54:09,799 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=574b09e98d501def4bb65b12ef4de07a, ASSIGN}] 2023-04-03 05:54:09,801 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=574b09e98d501def4bb65b12ef4de07a, ASSIGN 2023-04-03 05:54:09,802 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=574b09e98d501def4bb65b12ef4de07a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37397,1680501214665; forceNewPlan=false, retain=false 2023-04-03 05:54:09,953 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=574b09e98d501def4bb65b12ef4de07a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:09,953 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501249953"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501249953"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501249953"}]},"ts":"1680501249953"} 2023-04-03 05:54:09,955 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:54:10,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 05:54:10,110 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:10,110 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 574b09e98d501def4bb65b12ef4de07a, NAME => 'TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.', STARTKEY => '', ENDKEY => ''} 2023-04-03 05:54:10,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:10,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:54:10,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:10,111 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:10,112 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:10,114 DEBUG [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A 2023-04-03 05:54:10,114 DEBUG [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A 2023-04-03 05:54:10,114 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:54:10,114 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 574b09e98d501def4bb65b12ef4de07a columnFamilyName A 2023-04-03 05:54:10,115 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.HStore(310): Store=574b09e98d501def4bb65b12ef4de07a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:54:10,115 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:10,116 DEBUG [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B 2023-04-03 05:54:10,116 DEBUG [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B 2023-04-03 05:54:10,117 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:54:10,117 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 574b09e98d501def4bb65b12ef4de07a columnFamilyName B 2023-04-03 05:54:10,117 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.HStore(310): Store=574b09e98d501def4bb65b12ef4de07a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:54:10,118 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:10,119 DEBUG [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C 2023-04-03 05:54:10,119 DEBUG [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C 2023-04-03 05:54:10,119 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:54:10,119 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 574b09e98d501def4bb65b12ef4de07a columnFamilyName C 2023-04-03 05:54:10,119 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.HStore(310): Store=574b09e98d501def4bb65b12ef4de07a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:54:10,119 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:10,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:10,120 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:10,122 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 05:54:10,123 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:10,125 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 05:54:10,126 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 574b09e98d501def4bb65b12ef4de07a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64144894, jitterRate=-0.04416659474372864}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 05:54:10,126 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:10,126 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., pid=18, masterSystemTime=1680501250106 2023-04-03 05:54:10,128 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:10,128 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:10,128 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=574b09e98d501def4bb65b12ef4de07a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:10,128 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501250128"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680501250128"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680501250128"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680501250128"}]},"ts":"1680501250128"} 2023-04-03 05:54:10,131 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-03 05:54:10,131 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 in 174 msec 2023-04-03 05:54:10,132 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-03 05:54:10,132 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=574b09e98d501def4bb65b12ef4de07a, ASSIGN in 332 msec 2023-04-03 05:54:10,133 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 05:54:10,133 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501250133"}]},"ts":"1680501250133"} 2023-04-03 05:54:10,134 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 05:54:10,136 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 05:54:10,138 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1750 sec 2023-04-03 05:54:11,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-03 05:54:11,070 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-03 05:54:11,071 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x3ab850e7 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:11,077 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7053d0b0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:11,078 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:11,082 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53160, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:11,084 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 05:54:11,086 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57502, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 05:54:11,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 05:54:11,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 05:54:11,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:11,114 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 05:54:11,115 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 05:54:11,115 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 05:54:11,118 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-03 05:54:11,130 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=574b09e98d501def4bb65b12ef4de07a, REOPEN/MOVE}] 2023-04-03 05:54:11,131 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=574b09e98d501def4bb65b12ef4de07a, REOPEN/MOVE 2023-04-03 05:54:11,132 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=574b09e98d501def4bb65b12ef4de07a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:11,132 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501251132"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501251132"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501251132"}]},"ts":"1680501251132"} 2023-04-03 05:54:11,133 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:54:11,285 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:11,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 574b09e98d501def4bb65b12ef4de07a, disabling compactions & flushes 2023-04-03 05:54:11,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:11,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:11,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. after waiting 0 ms 2023-04-03 05:54:11,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:11,290 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-03 05:54:11,291 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:11,291 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:11,291 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 574b09e98d501def4bb65b12ef4de07a to self. 2023-04-03 05:54:11,292 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:11,293 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=574b09e98d501def4bb65b12ef4de07a, regionState=CLOSED 2023-04-03 05:54:11,293 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501251293"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501251293"}]},"ts":"1680501251293"} 2023-04-03 05:54:11,295 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-03 05:54:11,295 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 in 161 msec 2023-04-03 05:54:11,296 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=574b09e98d501def4bb65b12ef4de07a, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,37397,1680501214665; forceNewPlan=false, retain=true 2023-04-03 05:54:11,446 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=574b09e98d501def4bb65b12ef4de07a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:11,447 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501251446"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501251446"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501251446"}]},"ts":"1680501251446"} 2023-04-03 05:54:11,449 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:54:11,604 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:11,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 574b09e98d501def4bb65b12ef4de07a, NAME => 'TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.', STARTKEY => '', ENDKEY => ''} 2023-04-03 05:54:11,604 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:11,605 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:54:11,605 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:11,605 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:11,607 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:11,608 DEBUG [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A 2023-04-03 05:54:11,608 DEBUG [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A 2023-04-03 05:54:11,608 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:54:11,611 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 574b09e98d501def4bb65b12ef4de07a columnFamilyName A 2023-04-03 05:54:11,613 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.HStore(310): Store=574b09e98d501def4bb65b12ef4de07a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:54:11,613 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:11,614 DEBUG [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B 2023-04-03 05:54:11,614 DEBUG [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B 2023-04-03 05:54:11,614 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:54:11,615 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 574b09e98d501def4bb65b12ef4de07a columnFamilyName B 2023-04-03 05:54:11,615 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.HStore(310): Store=574b09e98d501def4bb65b12ef4de07a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:54:11,615 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:11,616 DEBUG [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C 2023-04-03 05:54:11,616 DEBUG [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C 2023-04-03 05:54:11,616 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:54:11,617 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 574b09e98d501def4bb65b12ef4de07a columnFamilyName C 2023-04-03 05:54:11,617 INFO [StoreOpener-574b09e98d501def4bb65b12ef4de07a-1] regionserver.HStore(310): Store=574b09e98d501def4bb65b12ef4de07a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:54:11,618 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:11,618 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:11,619 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:11,621 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 05:54:11,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:11,622 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 574b09e98d501def4bb65b12ef4de07a; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66373497, jitterRate=-0.01095782220363617}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 05:54:11,622 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:11,623 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., pid=23, masterSystemTime=1680501251600 2023-04-03 05:54:11,625 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:11,625 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:11,625 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=574b09e98d501def4bb65b12ef4de07a, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:11,625 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501251625"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680501251625"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680501251625"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680501251625"}]},"ts":"1680501251625"} 2023-04-03 05:54:11,628 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-03 05:54:11,628 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 in 177 msec 2023-04-03 05:54:11,629 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-03 05:54:11,629 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=574b09e98d501def4bb65b12ef4de07a, REOPEN/MOVE in 498 msec 2023-04-03 05:54:11,632 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-03 05:54:11,632 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 513 msec 2023-04-03 05:54:11,633 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 536 msec 2023-04-03 05:54:11,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-03 05:54:11,642 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x63ebf1e9 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:11,647 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@695af552, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:11,647 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x52d98fed to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:11,652 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52f29016, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:11,652 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x18496e17 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:11,655 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f13c14, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:11,656 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x25441895 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:11,660 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a5a81b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:11,661 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x40e0a3dc to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:11,664 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45e59be5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:11,664 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x2668d41d to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:11,669 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fbb4986, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:11,669 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x6b416c44 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:11,672 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b69850d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:11,673 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x204b5880 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:11,676 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@40883381, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:11,676 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x41173aec to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:11,680 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3be2e516, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:11,685 DEBUG [hconnection-0x2d2c0dcd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:11,686 DEBUG [hconnection-0x5a271569-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:11,687 DEBUG [hconnection-0x363ff606-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:11,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:11,687 DEBUG [hconnection-0x1a1e03e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:11,688 DEBUG [hconnection-0x4932fd5e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:11,690 DEBUG [hconnection-0x28a360da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:11,690 DEBUG [hconnection-0x176c4c2d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:11,691 DEBUG [hconnection-0x3e6d736b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:11,691 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53196, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:11,694 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53182, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:11,694 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53166, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:11,694 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53210, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:11,694 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53200, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:11,696 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53226, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:11,696 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53214, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:11,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:11,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:11,698 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53234, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:11,698 DEBUG [hconnection-0x7acf7cde-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:11,698 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:11,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:11,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:11,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:11,700 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:11,700 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:11,704 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:11,704 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:11,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:11,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:11,705 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53244, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:11,705 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:11,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:11,706 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:11,707 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:11,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:11,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:11,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:11,713 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:11,713 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:11,713 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:11,713 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:11,713 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:11,713 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:11,713 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:11,714 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:11,714 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:11,717 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-03 05:54:11,718 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:11,718 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:11,718 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:11,718 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:11,718 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:11,718 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:11,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:11,722 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:11,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501251735, max timestamp=1680501251745Num uniques 859; 2023-04-03 05:54:11,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:54:11,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680501251735, max timestamp=1680501251743Num uniques 859; 2023-04-03 05:54:11,756 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-03 05:54:11,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680501251735, max timestamp=1680501251743Num uniques 859; 2023-04-03 05:54:11,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:54:11,778 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403399a58839bf3456a8496740d1e07e7eb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403399a58839bf3456a8496740d1e07e7eb 2023-04-03 05:54:11,790 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680501251774, max timestamp=1680501251784Num uniques 1068; 2023-04-03 05:54:11,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-03 05:54:11,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680501251774, max timestamp=1680501251784Num uniques 1068; 2023-04-03 05:54:11,791 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-03 05:54:11,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680501251774, max timestamp=1680501251784Num uniques 1009; 2023-04-03 05:54:11,792 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=12268, suffix heap size=132904,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=268 2023-04-03 05:54:11,812 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=10, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/09304f0180b344e6adf748bd0f9f1ee7 2023-04-03 05:54:11,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:11,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501311809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:11,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680501251805, max timestamp=1680501251813Num uniques 1127; 2023-04-03 05:54:11,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680501251805, max timestamp=1680501251813Num uniques 1127; 2023-04-03 05:54:11,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-03 05:54:11,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-03 05:54:11,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680501251805, max timestamp=1680501251813Num uniques 1127; 2023-04-03 05:54:11,818 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-04-03 05:54:11,850 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680501251843, max timestamp=1680501251845Num uniques 1086; 2023-04-03 05:54:11,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-03 05:54:11,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680501251843, max timestamp=1680501251845Num uniques 1086; 2023-04-03 05:54:11,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-03 05:54:11,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680501251843, max timestamp=1680501251845Num uniques 1086; 2023-04-03 05:54:11,851 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=8516, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-03 05:54:11,854 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/53974844e3b04ea8ab99b3fdea024195 2023-04-03 05:54:11,885 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680501251862, max timestamp=1680501251881Num uniques 1045; 2023-04-03 05:54:11,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680501251862, max timestamp=1680501251881Num uniques 1045; 2023-04-03 05:54:11,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=245, cellSize=11220, totalHeapSize=25720, min timestamp=1680501251862, max timestamp=1680501251881Num uniques 1045; 2023-04-03 05:54:11,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-03 05:54:11,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-03 05:54:11,886 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=11220, suffix heap size=136648,new segment heap size=25720 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=245 2023-04-03 05:54:11,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680501251915, max timestamp=1680501251921Num uniques 1104; 2023-04-03 05:54:11,926 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=9340, suffix heap size=142784,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=204 2023-04-03 05:54:11,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680501251915, max timestamp=1680501251924Num uniques 1104; 2023-04-03 05:54:11,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-03 05:54:11,927 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680501251915, max timestamp=1680501251924Num uniques 1104; 2023-04-03 05:54:11,928 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50560, new segment data size=7050, suffix heap size=142784,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1104, new segment cells count=154 2023-04-03 05:54:11,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=413, cellSize=18908, totalHeapSize=43192, min timestamp=1680501251932, max timestamp=1680501251948Num uniques 1063; 2023-04-03 05:54:11,952 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=18908, suffix heap size=138520,new segment heap size=43192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=413 2023-04-03 05:54:11,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=413, cellSize=18908, totalHeapSize=43192, min timestamp=1680501251932, max timestamp=1680501251949Num uniques 1013; 2023-04-03 05:54:11,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=18908, suffix heap size=133320,new segment heap size=43192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=413 2023-04-03 05:54:11,953 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=413, cellSize=18908, totalHeapSize=43192, min timestamp=1680501251932, max timestamp=1680501251949Num uniques 1013; 2023-04-03 05:54:11,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=18908, suffix heap size=133320,new segment heap size=43192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=413 2023-04-03 05:54:11,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:11,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501311968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:11,977 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:11,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501311971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:11,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:11,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501311973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:11,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:11,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501311974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:11,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:11,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501311975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501312077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501312079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,080 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501312079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,081 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501312081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,082 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,082 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501312081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,275 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/ea39e43cfb1c432bac760ad659b27a89 2023-04-03 05:54:12,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501312281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,283 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/09304f0180b344e6adf748bd0f9f1ee7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/09304f0180b344e6adf748bd0f9f1ee7 2023-04-03 05:54:12,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501312283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501312283, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501312284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,285 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501312284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,290 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/09304f0180b344e6adf748bd0f9f1ee7, entries=150, sequenceid=10, filesize=25.1 K 2023-04-03 05:54:12,291 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/53974844e3b04ea8ab99b3fdea024195 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/53974844e3b04ea8ab99b3fdea024195 2023-04-03 05:54:12,304 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/53974844e3b04ea8ab99b3fdea024195, entries=150, sequenceid=10, filesize=11.6 K 2023-04-03 05:54:12,306 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/ea39e43cfb1c432bac760ad659b27a89 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/ea39e43cfb1c432bac760ad659b27a89 2023-04-03 05:54:12,314 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/ea39e43cfb1c432bac760ad659b27a89, entries=150, sequenceid=10, filesize=11.6 K 2023-04-03 05:54:12,315 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=167.72 KB/171750 for 574b09e98d501def4bb65b12ef4de07a in 600ms, sequenceid=10, compaction requested=false 2023-04-03 05:54:12,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:12,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:12,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:12,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:12,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,316 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:12,316 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:12,321 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:12,322 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:12,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:12,322 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:12,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:12,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:12,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:12,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b7740d0[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:12,324 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:12,324 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:12,328 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:12,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:12,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:12,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:12,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:12,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:12,329 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,329 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:12,329 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:12,329 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:12,329 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:12,330 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:12,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:12,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:12,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:12,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:12,333 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:12,333 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:12,333 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:12,333 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:12,333 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:12,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:12,337 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:12,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:12,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:12,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:12,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:12,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:12,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:12,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:12,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:12,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:12,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:12,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:12,340 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:12,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:12,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:12,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,344 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:12,344 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:12,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:12,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:12,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:12,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:12,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:12,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:12,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:12,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:12,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:12,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:12,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:12,345 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:12,345 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:12,345 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:12,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:12,445 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:12,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:12,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:12,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:12,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:12,464 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:12,464 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:12,464 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:12,464 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:12,465 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:12,465 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:12,466 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:12,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:12,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:12,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:12,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:12,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:12,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:12,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:12,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:12,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:12,480 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:12,480 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:12,480 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:12,480 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:12,480 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=167.72 KB heapSize=460.78 KB 2023-04-03 05:54:12,481 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:12,481 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:12,481 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:12,481 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:12,481 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:12,481 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:12,523 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304032867b16cbae841e3bdb5e6820e093bd2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032867b16cbae841e3bdb5e6820e093bd2 2023-04-03 05:54:12,540 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=150, memsize=55.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/f3211418e47e4182ade9ade3914f787c 2023-04-03 05:54:12,587 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:12,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:12,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,593 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501312589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501312589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501312593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,597 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501312593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,598 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501312594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501312694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501312698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501312699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,699 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501312699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,899 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501312897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501312901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501312901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:12,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501312902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:12,992 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/0e8cd1f5284546b58912fd137a654382 2023-04-03 05:54:13,023 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=55.91 KB at sequenceid=150 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/6e5e0116849d4f1fa51a78fa8cb1d9b6 2023-04-03 05:54:13,031 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/f3211418e47e4182ade9ade3914f787c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/f3211418e47e4182ade9ade3914f787c 2023-04-03 05:54:13,038 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/f3211418e47e4182ade9ade3914f787c, entries=150, sequenceid=150, filesize=25.3 K 2023-04-03 05:54:13,040 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/0e8cd1f5284546b58912fd137a654382 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0e8cd1f5284546b58912fd137a654382 2023-04-03 05:54:13,061 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0e8cd1f5284546b58912fd137a654382, entries=150, sequenceid=150, filesize=11.8 K 2023-04-03 05:54:13,062 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/6e5e0116849d4f1fa51a78fa8cb1d9b6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6e5e0116849d4f1fa51a78fa8cb1d9b6 2023-04-03 05:54:13,091 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6e5e0116849d4f1fa51a78fa8cb1d9b6, entries=150, sequenceid=150, filesize=11.8 K 2023-04-03 05:54:13,092 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~167.72 KB/171750, heapSize ~460.73 KB/471792, currentSize=26.84 KB/27480 for 574b09e98d501def4bb65b12ef4de07a in 611ms, sequenceid=150, compaction requested=false 2023-04-03 05:54:13,092 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:13,092 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:13,092 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:13,094 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:13,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,095 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:13,096 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,096 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:13,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:13,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32fc3fa4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:13,096 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:13,096 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,098 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,098 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:13,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,099 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:13,099 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:13,099 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:13,105 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,105 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:13,105 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:13,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:13,105 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:13,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:13,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:13,110 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:54:13,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:13,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:13,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:13,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:13,110 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:13,110 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:13,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:13,112 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:13,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:13,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:13,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:13,112 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:13,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:13,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:13,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,115 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:13,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,117 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,119 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:13,119 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:13,119 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:13,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:13,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:13,120 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:13,120 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:13,120 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:13,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:13,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:13,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:13,159 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403ad64bd4e443f455f8a6daa4413ce55cb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ad64bd4e443f455f8a6daa4413ce55cb 2023-04-03 05:54:13,170 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=160, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/8c90e4cec55c4aaeaf70bc910cef8ecf 2023-04-03 05:54:13,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501253169, max timestamp=1680501253184Num uniques 859; 2023-04-03 05:54:13,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501253169, max timestamp=1680501253184Num uniques 859; 2023-04-03 05:54:13,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:13,186 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:13,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501253169, max timestamp=1680501253184Num uniques 859; 2023-04-03 05:54:13,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:13,220 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:13,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501313214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:13,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680501253207, max timestamp=1680501253219Num uniques 1018; 2023-04-03 05:54:13,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-04-03 05:54:13,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680501253207, max timestamp=1680501253219Num uniques 1018; 2023-04-03 05:54:13,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=368, cellSize=16848, totalHeapSize=38512, min timestamp=1680501253207, max timestamp=1680501253219Num uniques 1018; 2023-04-03 05:54:13,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-04-03 05:54:13,222 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=16848, suffix heap size=133840,new segment heap size=38512 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=368 2023-04-03 05:54:13,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:13,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:13,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:13,227 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:13,228 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:13,228 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:13,228 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:13,228 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,228 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,229 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:13,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:13,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:13,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,231 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:13,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:13,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,231 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:13,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:13,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:13,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:13,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:13,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:13,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:13,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:13,241 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:13,241 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:13,241 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:13,241 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:13,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:13,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:13,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:13,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:13,244 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:13,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:13,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:13,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:13,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:13,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62906ac7[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:13,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:13,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,249 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,249 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:13,249 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:13,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:13,249 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:13,254 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501313248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501313252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501313253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,257 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:13,257 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:13,257 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:13,257 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:13,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:13,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501313255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:13,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:13,263 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:13,263 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:13,263 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:13,263 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:13,263 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:13,268 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,268 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:13,268 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:13,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:13,268 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,273 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,276 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:13,277 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,277 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:13,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:13,277 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:13,277 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:13,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:13,277 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,277 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,277 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,277 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,277 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,277 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:13,277 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:13,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:13,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:13,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501313324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,355 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501313355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501313358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501313358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501313362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,377 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:13,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:13,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:13,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:13,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:13,400 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:13,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:13,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:13,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:13,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,402 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:13,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:13,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:13,402 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:13,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:13,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:13,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:13,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:13,416 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:13,417 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:13,417 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:13,417 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:13,417 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:13,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:13,420 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:13,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:13,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:13,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:13,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58ffe7a7[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:13,425 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:13,425 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,427 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,427 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:13,427 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:13,427 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:13,428 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,428 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:13,430 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:13,430 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:13,430 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,431 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:13,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:13,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:13,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:13,440 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:13,441 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:13,441 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:13,441 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:13,441 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:13,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:13,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:13,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:13,444 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:13,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,446 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:13,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:13,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:13,450 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:13,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:13,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:13,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:13,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,450 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:13,451 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:13,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:13,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:13,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:13,482 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 05:54:13,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501313530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,551 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:13,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:13,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:13,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:13,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:13,556 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:13,556 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:13,556 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:13,556 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:13,556 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,557 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,558 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,558 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:13,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:13,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,559 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,559 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:13,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:13,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:13,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:13,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:13,560 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:13,560 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:13,560 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:13,560 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:13,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,560 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501313559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,561 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:13,561 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:13,564 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,564 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:13,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:13,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501313562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501313563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:13,566 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,566 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:13,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:13,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@693f4da8[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:13,566 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:13,566 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,568 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,568 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:13,568 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:13,568 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:13,568 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,568 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:13,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501313567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,569 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:13,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:13,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:13,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:13,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:13,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:13,572 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:13,572 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:13,572 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:13,572 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:13,572 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:13,580 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,580 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:13,580 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:13,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:13,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:13,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,584 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/b88813d13e364bfbafb095650ada1322 2023-04-03 05:54:13,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:13,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:13,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:13,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:13,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,595 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:13,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:13,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:13,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:13,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:13,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:13,595 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:13,596 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:13,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:13,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,596 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:13,596 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:13,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:13,597 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:13,614 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=160 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/8fd29ddc81474b279e4ba6e70fd606fe 2023-04-03 05:54:13,624 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/8c90e4cec55c4aaeaf70bc910cef8ecf as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/8c90e4cec55c4aaeaf70bc910cef8ecf 2023-04-03 05:54:13,635 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/8c90e4cec55c4aaeaf70bc910cef8ecf, entries=200, sequenceid=160, filesize=32.1 K 2023-04-03 05:54:13,638 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/b88813d13e364bfbafb095650ada1322 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/b88813d13e364bfbafb095650ada1322 2023-04-03 05:54:13,644 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/b88813d13e364bfbafb095650ada1322, entries=150, sequenceid=160, filesize=11.8 K 2023-04-03 05:54:13,645 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/8fd29ddc81474b279e4ba6e70fd606fe as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/8fd29ddc81474b279e4ba6e70fd606fe 2023-04-03 05:54:13,649 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/8fd29ddc81474b279e4ba6e70fd606fe, entries=150, sequenceid=160, filesize=11.8 K 2023-04-03 05:54:13,650 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 574b09e98d501def4bb65b12ef4de07a in 540ms, sequenceid=160, compaction requested=true 2023-04-03 05:54:13,650 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:13,650 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:13,650 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:13,650 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:13,650 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:13,651 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:13,652 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84528 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:13,652 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/A is initiating minor compaction (all files) 2023-04-03 05:54:13,652 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/A in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:13,652 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/09304f0180b344e6adf748bd0f9f1ee7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/f3211418e47e4182ade9ade3914f787c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/8c90e4cec55c4aaeaf70bc910cef8ecf] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=82.5 K 2023-04-03 05:54:13,652 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:13,653 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/B is initiating minor compaction (all files) 2023-04-03 05:54:13,653 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/B in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:13,653 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/53974844e3b04ea8ab99b3fdea024195, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0e8cd1f5284546b58912fd137a654382, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/b88813d13e364bfbafb095650ada1322] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=35.1 K 2023-04-03 05:54:13,653 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 09304f0180b344e6adf748bd0f9f1ee7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680501251713 2023-04-03 05:54:13,653 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 53974844e3b04ea8ab99b3fdea024195, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680501251713 2023-04-03 05:54:13,653 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting f3211418e47e4182ade9ade3914f787c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680501251969 2023-04-03 05:54:13,654 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0e8cd1f5284546b58912fd137a654382, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680501251969 2023-04-03 05:54:13,654 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8c90e4cec55c4aaeaf70bc910cef8ecf, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680501252589 2023-04-03 05:54:13,654 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b88813d13e364bfbafb095650ada1322, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680501252589 2023-04-03 05:54:13,669 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#B#compaction#130 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:13,676 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#A#compaction#129 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:13,693 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/98e3b72ab2e14cd1843762077c3416cf as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/98e3b72ab2e14cd1843762077c3416cf 2023-04-03 05:54:13,696 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:13,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:13,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:13,701 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/B of 574b09e98d501def4bb65b12ef4de07a into 98e3b72ab2e14cd1843762077c3416cf(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:13,702 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:13,702 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/B, priority=13, startTime=1680501253650; duration=0sec 2023-04-03 05:54:13,702 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:13,702 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:13,703 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35958 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:13,703 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/C is initiating minor compaction (all files) 2023-04-03 05:54:13,703 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/C in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:13,703 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/ea39e43cfb1c432bac760ad659b27a89, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6e5e0116849d4f1fa51a78fa8cb1d9b6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/8fd29ddc81474b279e4ba6e70fd606fe] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=35.1 K 2023-04-03 05:54:13,704 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ea39e43cfb1c432bac760ad659b27a89, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680501251713 2023-04-03 05:54:13,704 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6e5e0116849d4f1fa51a78fa8cb1d9b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=150, earliestPutTs=1680501251969 2023-04-03 05:54:13,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:13,705 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 8fd29ddc81474b279e4ba6e70fd606fe, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680501252589 2023-04-03 05:54:13,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:13,705 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:13,705 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:13,705 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:13,705 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:13,706 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,706 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,707 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:13,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:13,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:13,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,708 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:13,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:13,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:13,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:13,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:13,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:13,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:13,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:13,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:13,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:13,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:13,711 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:13,711 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:13,711 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=140.89 KB heapSize=385.31 KB 2023-04-03 05:54:13,712 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:13,712 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:13,713 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#C#compaction#131 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:13,718 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:13,719 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:13,719 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:13,719 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:13,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:13,836 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:13,858 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,858 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501313855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,867 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501313864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501313866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501313868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,871 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501313870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:13,960 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:13,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501313959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,125 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/cb68b13b3d42464f873b5e481f8dfb51 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/cb68b13b3d42464f873b5e481f8dfb51 2023-04-03 05:54:14,130 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/A of 574b09e98d501def4bb65b12ef4de07a into cb68b13b3d42464f873b5e481f8dfb51(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:14,130 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:14,130 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/A, priority=13, startTime=1680501253650; duration=0sec 2023-04-03 05:54:14,130 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:14,136 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/05b7a3e562c84782b3d390ebf4b6511b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/05b7a3e562c84782b3d390ebf4b6511b 2023-04-03 05:54:14,146 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403053baaed87474fa9b18986be9732f19b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403053baaed87474fa9b18986be9732f19b 2023-04-03 05:54:14,147 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/C of 574b09e98d501def4bb65b12ef4de07a into 05b7a3e562c84782b3d390ebf4b6511b(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:14,147 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:14,148 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/C, priority=13, startTime=1680501253651; duration=0sec 2023-04-03 05:54:14,148 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:14,163 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:14,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501314163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,180 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=212, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/c774e64471cb45b2b4456053ea355cb2 2023-04-03 05:54:14,230 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/a6fa374e48224681ab06405acd468114 2023-04-03 05:54:14,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:14,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501314370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:14,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501314372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:14,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501314374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:14,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501314376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:14,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501314465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,650 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=212 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/81238b0c4b214aa7956806660318d0da 2023-04-03 05:54:14,655 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/c774e64471cb45b2b4456053ea355cb2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/c774e64471cb45b2b4456053ea355cb2 2023-04-03 05:54:14,660 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/c774e64471cb45b2b4456053ea355cb2, entries=150, sequenceid=212, filesize=25.3 K 2023-04-03 05:54:14,661 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/a6fa374e48224681ab06405acd468114 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/a6fa374e48224681ab06405acd468114 2023-04-03 05:54:14,670 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/a6fa374e48224681ab06405acd468114, entries=150, sequenceid=212, filesize=11.8 K 2023-04-03 05:54:14,671 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/81238b0c4b214aa7956806660318d0da as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/81238b0c4b214aa7956806660318d0da 2023-04-03 05:54:14,675 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/81238b0c4b214aa7956806660318d0da, entries=150, sequenceid=212, filesize=11.8 K 2023-04-03 05:54:14,676 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~385.27 KB/394512, currentSize=46.96 KB/48090 for 574b09e98d501def4bb65b12ef4de07a in 965ms, sequenceid=212, compaction requested=false 2023-04-03 05:54:14,676 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:14,676 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:14,676 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:14,676 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:14,676 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,677 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:14,677 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:14,679 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,679 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:14,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:14,679 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:14,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:14,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:14,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:14,681 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:14,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3aed5e34[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:14,681 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:14,683 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:14,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:14,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:14,683 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:14,683 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:14,683 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:14,683 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,684 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:14,685 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:14,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:14,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:14,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:14,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:14,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:14,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:14,688 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:14,688 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:14,688 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:14,688 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:14,688 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:14,690 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:14,690 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:14,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:14,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:14,690 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:14,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:14,691 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:14,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:14,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:14,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:14,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:14,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,695 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:14,695 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,695 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:14,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:14,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:14,695 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:14,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:14,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:14,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:14,695 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:14,696 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:14,696 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,696 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:14,696 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:14,696 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:14,696 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:14,696 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:14,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:14,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:14,796 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:14,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:14,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:14,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:14,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:14,801 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:14,801 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:14,801 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:14,801 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:14,801 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:14,801 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:14,804 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:14,804 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:14,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:14,804 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:14,804 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:14,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:14,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:14,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:14,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:14,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:14,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:14,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:14,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:14,824 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:14,824 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:14,824 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:54:14,824 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:14,824 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:14,825 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:14,825 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:14,825 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:14,825 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:14,854 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304030e636430e4ef46a1b893870ff88d7419 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030e636430e4ef46a1b893870ff88d7419 2023-04-03 05:54:14,883 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=224, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/7991607bb4894adeb98cf9133c4367d0 2023-04-03 05:54:14,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:14,969 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:15,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501255008, max timestamp=1680501255023Num uniques 859; 2023-04-03 05:54:15,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:54:15,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501255008, max timestamp=1680501255023Num uniques 859; 2023-04-03 05:54:15,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501255008, max timestamp=1680501255023Num uniques 859; 2023-04-03 05:54:15,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:54:15,025 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:54:15,075 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501315072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501315176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,302 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/f16460db815e4f5189e873aa56655ac8 2023-04-03 05:54:15,314 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/d284eee7f26d44a3ae5e77e29f4eaa18 2023-04-03 05:54:15,318 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/7991607bb4894adeb98cf9133c4367d0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/7991607bb4894adeb98cf9133c4367d0 2023-04-03 05:54:15,322 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/7991607bb4894adeb98cf9133c4367d0, entries=150, sequenceid=224, filesize=25.3 K 2023-04-03 05:54:15,322 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.StoreScanner(953): StoreScanner already has the close lock. There is no need to updateReaders 2023-04-03 05:54:15,323 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/f16460db815e4f5189e873aa56655ac8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/f16460db815e4f5189e873aa56655ac8 2023-04-03 05:54:15,327 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/f16460db815e4f5189e873aa56655ac8, entries=150, sequenceid=224, filesize=11.8 K 2023-04-03 05:54:15,328 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/d284eee7f26d44a3ae5e77e29f4eaa18 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/d284eee7f26d44a3ae5e77e29f4eaa18 2023-04-03 05:54:15,332 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/d284eee7f26d44a3ae5e77e29f4eaa18, entries=150, sequenceid=224, filesize=11.8 K 2023-04-03 05:54:15,333 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 574b09e98d501def4bb65b12ef4de07a in 509ms, sequenceid=224, compaction requested=true 2023-04-03 05:54:15,333 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:15,333 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:15,333 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:15,335 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,335 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,335 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,335 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:15,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b4addfc[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:15,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:15,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,338 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:15,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:15,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:15,338 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,338 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:15,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:15,340 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:15,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:15,343 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:15,343 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:15,343 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:15,343 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:15,343 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:15,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,346 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,348 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,350 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,350 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,350 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,350 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:15,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:15,350 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:15,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:15,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:15,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:15,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:15,350 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:15,350 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:15,350 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,351 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:15,351 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,351 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,351 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:15,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:15,377 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-03 05:54:15,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:15,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:15,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:15,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:15,377 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:15,377 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:15,383 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501315381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501315382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501315382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501315383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,387 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501315383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,451 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:15,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:15,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:15,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:15,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:15,454 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:15,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:15,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:15,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:15,455 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,455 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,456 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,456 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:15,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:15,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:15,456 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,456 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,457 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:15,458 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:15,458 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:15,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:15,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:15,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:15,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:15,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:15,458 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:15,458 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:15,458 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:15,458 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:15,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:15,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:15,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:15,459 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:15,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,460 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,460 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,460 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:15,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cafdbed[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:15,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:15,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,464 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,464 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:15,464 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:15,464 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:15,464 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,464 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:15,465 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:15,465 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:15,465 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,466 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:15,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:15,468 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:15,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:15,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:15,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:15,468 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:15,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:15,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:15,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:15,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,471 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:15,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:15,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:15,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:15,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:15,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,478 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:15,478 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:15,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:15,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:15,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:15,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,487 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501315484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501315485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501315487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,488 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501315488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,579 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:15,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:15,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:15,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:15,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:15,582 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:15,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:15,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:15,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:15,583 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,583 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,584 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:15,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:15,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,585 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,585 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:15,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:15,586 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:15,586 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:15,586 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:15,586 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:15,586 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:15,586 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:15,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:15,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,588 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,588 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,588 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:15,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6e9bdd[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:15,590 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:15,590 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,591 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:15,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:15,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:15,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:15,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:15,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:15,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:15,596 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:15,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:15,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:15,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:15,597 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:15,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:15,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:15,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,598 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,600 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,602 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,602 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,602 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:15,602 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:15,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:15,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:15,602 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:15,603 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,603 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:15,603 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:15,603 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:15,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:15,603 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:15,603 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,603 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:15,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:15,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501315688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501315689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501315689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501315690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,703 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:15,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:15,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:15,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:15,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:15,706 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:15,706 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:15,706 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:15,706 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:15,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,708 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:15,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:15,708 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,709 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,709 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:15,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:15,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:15,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:15,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:15,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:15,710 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:15,710 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:15,710 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:15,710 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:15,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:15,712 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,712 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:15,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3fa274fa[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:15,714 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:15,714 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,716 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:15,716 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:15,716 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,716 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:15,716 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:15,717 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:15,717 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,717 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,717 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:15,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:15,720 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:15,720 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:15,720 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:15,720 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:15,720 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:15,722 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,722 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,722 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:15,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,723 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,724 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,727 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:15,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:15,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:15,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:15,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:15,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:15,728 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:15,728 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:15,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:15,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:15,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:15,799 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304030cfabe56ba2544c0a88522055ea2e846 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030cfabe56ba2544c0a88522055ea2e846 2023-04-03 05:54:15,810 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=261, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/2f36ca9b1278489283ef671bc4198045 2023-04-03 05:54:15,828 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:15,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:15,830 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/cbb5ca7dfe29464f88d826b0399bf7c5 2023-04-03 05:54:15,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:15,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:15,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:15,833 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:15,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:15,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:15,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:15,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,834 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:15,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:15,835 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,835 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,835 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:15,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:15,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:15,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:15,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:15,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:15,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:15,844 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:15,844 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:15,844 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:15,844 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:15,844 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:15,844 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:15,845 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:15,845 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:15,845 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,848 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:15,848 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:15,849 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,850 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,850 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,850 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:15,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@328afbcf[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:15,852 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:15,852 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,853 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,853 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:15,854 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:15,854 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:15,854 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,854 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:15,860 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:15,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:15,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:15,864 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:15,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:15,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:15,865 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:15,865 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:15,866 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,866 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:15,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:15,867 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,869 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,874 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,874 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,874 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,874 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,874 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:15,874 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:15,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:15,875 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:15,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:15,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:15,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:15,875 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:15,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:15,875 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:15,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,975 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:15,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:15,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:15,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:15,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:15,979 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:15,979 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:15,979 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:15,979 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:15,979 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,979 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,981 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:15,981 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:15,982 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,982 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:15,982 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:15,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:15,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:15,983 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:15,983 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:15,983 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:15,983 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:15,983 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:15,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:15,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:15,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:15,985 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,985 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,986 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,986 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:15,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30783ec0[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:15,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:15,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,988 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:15,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:15,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:15,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:15,988 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,989 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:15,990 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,990 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:15,990 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,990 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53200 deadline: 1680501315990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:15,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:15,994 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:15,994 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:15,994 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:15,994 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:15,994 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:15,997 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,997 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:15,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,997 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:15,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:15,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:15,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501315997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:15,998 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:15,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:15,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:15,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:15,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501315998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501315998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,002 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:16,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:16,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:16,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,002 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:16,002 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:16,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,103 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:16,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:16,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:16,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:16,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:16,106 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:16,106 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:16,106 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:16,106 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:16,106 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,107 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,108 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,108 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,108 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,108 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:16,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:16,109 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:16,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:16,109 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:16,109 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:16,109 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:16,109 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:16,109 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:16,109 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,109 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:16,110 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:16,110 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:16,110 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:16,111 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,111 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,112 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,112 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:16,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd3eb61[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:16,113 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:16,113 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,115 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:16,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:16,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:16,115 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,115 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:16,117 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:16,117 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,117 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,117 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:16,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:16,119 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:16,119 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:16,119 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:16,119 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:16,119 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:16,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,121 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,121 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,122 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,124 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,124 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,124 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,124 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:16,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:16,124 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:16,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,124 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,125 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:16,126 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:16,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,226 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:16,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:16,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:16,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:16,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:16,230 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:16,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:16,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:16,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:16,231 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,231 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,233 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,233 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,233 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:16,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:16,235 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:16,235 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,235 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:16,235 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:16,235 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:16,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:16,237 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,237 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:16,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77a5f45[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:16,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:16,239 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,239 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:16,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:16,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:16,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:16,241 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:16,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:16,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:16,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:16,244 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:16,244 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:16,244 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:16,244 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:16,244 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:16,247 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,247 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,247 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,248 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:16,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:16,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,251 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:16,252 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:16,252 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:16,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,274 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/f630ed6d722d45ca8add323c9bef0f81 2023-04-03 05:54:16,282 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/2f36ca9b1278489283ef671bc4198045 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/2f36ca9b1278489283ef671bc4198045 2023-04-03 05:54:16,287 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/2f36ca9b1278489283ef671bc4198045, entries=200, sequenceid=261, filesize=32.3 K 2023-04-03 05:54:16,289 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/cbb5ca7dfe29464f88d826b0399bf7c5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/cbb5ca7dfe29464f88d826b0399bf7c5 2023-04-03 05:54:16,296 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/cbb5ca7dfe29464f88d826b0399bf7c5, entries=150, sequenceid=261, filesize=11.9 K 2023-04-03 05:54:16,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/f630ed6d722d45ca8add323c9bef0f81 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f630ed6d722d45ca8add323c9bef0f81 2023-04-03 05:54:16,303 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f630ed6d722d45ca8add323c9bef0f81, entries=150, sequenceid=261, filesize=11.9 K 2023-04-03 05:54:16,304 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 574b09e98d501def4bb65b12ef4de07a in 927ms, sequenceid=261, compaction requested=true 2023-04-03 05:54:16,304 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:16,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:16,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:16,304 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:16,304 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:16,304 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:16,306 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110827 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:16,306 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/A is initiating minor compaction (all files) 2023-04-03 05:54:16,307 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/A in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,307 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/cb68b13b3d42464f873b5e481f8dfb51, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/c774e64471cb45b2b4456053ea355cb2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/7991607bb4894adeb98cf9133c4367d0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/2f36ca9b1278489283ef671bc4198045] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=108.2 K 2023-04-03 05:54:16,307 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting cb68b13b3d42464f873b5e481f8dfb51, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680501252589 2023-04-03 05:54:16,307 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48347 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:16,307 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/B is initiating minor compaction (all files) 2023-04-03 05:54:16,308 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/B in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,308 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/98e3b72ab2e14cd1843762077c3416cf, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/a6fa374e48224681ab06405acd468114, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/f16460db815e4f5189e873aa56655ac8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/cbb5ca7dfe29464f88d826b0399bf7c5] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=47.2 K 2023-04-03 05:54:16,308 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting c774e64471cb45b2b4456053ea355cb2, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1680501253246 2023-04-03 05:54:16,308 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 98e3b72ab2e14cd1843762077c3416cf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680501252589 2023-04-03 05:54:16,309 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 7991607bb4894adeb98cf9133c4367d0, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680501253842 2023-04-03 05:54:16,309 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a6fa374e48224681ab06405acd468114, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1680501253246 2023-04-03 05:54:16,309 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 2f36ca9b1278489283ef671bc4198045, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680501255057 2023-04-03 05:54:16,309 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting f16460db815e4f5189e873aa56655ac8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680501253842 2023-04-03 05:54:16,310 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting cbb5ca7dfe29464f88d826b0399bf7c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680501255057 2023-04-03 05:54:16,322 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#B#compaction#141 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:16,348 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#A#compaction#142 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:16,352 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:16,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:16,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:16,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:16,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:16,361 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:16,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:16,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:16,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:16,362 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,362 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,364 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,364 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,364 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,364 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:16,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,365 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,365 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:16,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:16,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:16,366 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:16,366 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:16,366 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:16,368 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:16,368 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:16,368 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,368 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:16,369 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-03 05:54:16,370 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:16,370 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:16,371 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:16,371 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:16,371 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:16,371 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:16,372 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/a7aa188159074109882955611e9f9717 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/a7aa188159074109882955611e9f9717 2023-04-03 05:54:16,379 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/B of 574b09e98d501def4bb65b12ef4de07a into a7aa188159074109882955611e9f9717(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:16,379 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:16,379 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/B, priority=12, startTime=1680501256304; duration=0sec 2023-04-03 05:54:16,379 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:16,379 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:16,382 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48347 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:16,382 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/C is initiating minor compaction (all files) 2023-04-03 05:54:16,382 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/C in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,382 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/05b7a3e562c84782b3d390ebf4b6511b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/81238b0c4b214aa7956806660318d0da, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/d284eee7f26d44a3ae5e77e29f4eaa18, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f630ed6d722d45ca8add323c9bef0f81] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=47.2 K 2023-04-03 05:54:16,382 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 05b7a3e562c84782b3d390ebf4b6511b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=160, earliestPutTs=1680501252589 2023-04-03 05:54:16,384 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 81238b0c4b214aa7956806660318d0da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=212, earliestPutTs=1680501253246 2023-04-03 05:54:16,384 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting d284eee7f26d44a3ae5e77e29f4eaa18, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1680501253842 2023-04-03 05:54:16,385 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting f630ed6d722d45ca8add323c9bef0f81, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680501255057 2023-04-03 05:54:16,397 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#C#compaction#144 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:16,403 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/b5595f86b1374f70b2e36b00405b6fc5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/b5595f86b1374f70b2e36b00405b6fc5 2023-04-03 05:54:16,408 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/A of 574b09e98d501def4bb65b12ef4de07a into b5595f86b1374f70b2e36b00405b6fc5(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:16,408 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:16,408 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/A, priority=12, startTime=1680501256304; duration=0sec 2023-04-03 05:54:16,408 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:16,410 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304036396070ed1434e0987fbd2f342cf55a3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036396070ed1434e0987fbd2f342cf55a3 2023-04-03 05:54:16,421 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/76c33087cd424a3f88e7f87642ea6a91 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/76c33087cd424a3f88e7f87642ea6a91 2023-04-03 05:54:16,430 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/C of 574b09e98d501def4bb65b12ef4de07a into 76c33087cd424a3f88e7f87642ea6a91(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:16,430 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:16,430 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/C, priority=12, startTime=1680501256304; duration=0sec 2023-04-03 05:54:16,430 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:16,436 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=270, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/b9fe9df10cf04a01981686ada3454325 2023-04-03 05:54:16,449 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/0f679bef98a746de8cc701df679af2f8 2023-04-03 05:54:16,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:16,495 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:16,509 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/84f688ead23e47ae823c3eb637ec9932 2023-04-03 05:54:16,515 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/b9fe9df10cf04a01981686ada3454325 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/b9fe9df10cf04a01981686ada3454325 2023-04-03 05:54:16,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680501256509, max timestamp=1680501256516Num uniques 859; 2023-04-03 05:54:16,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680501256509, max timestamp=1680501256516Num uniques 859; 2023-04-03 05:54:16,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:54:16,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:54:16,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680501256509, max timestamp=1680501256516Num uniques 859; 2023-04-03 05:54:16,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:54:16,524 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/b9fe9df10cf04a01981686ada3454325, entries=150, sequenceid=270, filesize=25.4 K 2023-04-03 05:54:16,526 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/0f679bef98a746de8cc701df679af2f8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0f679bef98a746de8cc701df679af2f8 2023-04-03 05:54:16,532 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0f679bef98a746de8cc701df679af2f8, entries=150, sequenceid=270, filesize=11.9 K 2023-04-03 05:54:16,533 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/84f688ead23e47ae823c3eb637ec9932 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/84f688ead23e47ae823c3eb637ec9932 2023-04-03 05:54:16,544 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/84f688ead23e47ae823c3eb637ec9932, entries=150, sequenceid=270, filesize=11.9 K 2023-04-03 05:54:16,545 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=134.18 KB/137400 for 574b09e98d501def4bb65b12ef4de07a in 176ms, sequenceid=270, compaction requested=false 2023-04-03 05:54:16,545 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:16,545 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,545 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:16,545 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:16,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:16,546 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-03 05:54:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:16,546 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:16,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:16,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:16,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:16,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:16,547 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:16,547 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:16,551 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,551 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,552 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,552 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:16,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@624b88e6[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:16,554 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:16,554 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,555 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501316552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,556 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,556 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:16,556 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:16,556 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:16,556 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,557 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:16,558 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:16,558 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,558 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:16,558 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501316554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,560 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501316555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,561 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501316555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:16,565 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:16,565 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:16,565 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:16,565 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:16,565 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:16,565 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:16,566 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,566 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,566 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,569 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,572 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:16,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:16,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:16,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,572 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:16,572 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:16,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,573 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,573 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,573 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,578 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403289f3c0a2795488ca86a3ecfa541eab1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403289f3c0a2795488ca86a3ecfa541eab1 2023-04-03 05:54:16,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501316656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,663 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501316659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501316661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501316662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,673 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:16,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:16,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:16,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:16,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:16,676 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:16,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:16,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:16,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:16,677 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,677 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,678 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,678 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,678 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,678 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:16,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:16,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,679 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:16,679 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:16,679 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:16,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:16,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:16,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:16,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:16,682 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,682 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,682 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,682 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:16,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:16,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13b43451[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:16,684 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:16,684 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,685 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:16,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:16,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:16,685 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,685 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:16,688 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,688 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:16,688 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,688 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:16,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:16,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:16,691 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:16,691 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:16,691 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:16,691 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:16,691 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:16,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,697 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:16,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:16,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:16,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,697 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:16,697 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:16,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,798 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:16,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:16,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:16,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:16,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:16,801 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:16,801 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:16,801 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:16,801 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:16,802 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,802 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,803 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,803 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,803 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,803 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:16,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:16,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:16,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:16,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:16,805 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:16,805 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:16,805 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:16,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:16,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:16,806 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,806 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:16,806 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:16,806 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:16,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:16,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:16,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:16,809 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,809 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,809 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:16,809 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:16,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27085da5[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:16,811 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:16,811 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,812 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:16,812 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:16,812 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:16,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:16,813 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,814 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:16,814 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,814 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:16,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:16,816 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:16,816 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:16,816 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:16,816 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:16,816 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:16,818 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,818 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,819 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,820 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:16,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:16,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,825 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,825 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,824 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,825 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:16,825 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:16,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,825 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:16,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501316858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501316865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501316867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:16,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501316867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,925 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:16,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:16,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:16,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:16,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:16,929 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:16,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:16,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:16,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:16,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,930 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,931 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,932 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,932 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:16,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:16,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:16,940 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:16,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:16,940 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:16,941 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,941 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:16,941 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:16,941 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:16,941 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:16,943 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,943 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,943 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,944 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:16,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725dae76[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:16,946 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:16,946 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,947 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:16,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:16,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:16,952 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:16,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:16,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:16,955 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:16,956 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:16,956 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:16,956 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:16,956 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:16,958 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,958 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:16,958 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:16,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,965 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:16,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:16,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:16,969 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:16,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:16,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:16,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:16,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:16,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:16,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:16,970 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:16,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:16,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:16,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:16,970 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:16,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:16,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:16,986 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=309, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/4d56ed6499e243e38a0ce3d159a72441 2023-04-03 05:54:16,998 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/40ee4ab0a6954716ae520a4cd68f3878 2023-04-03 05:54:17,023 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=309 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/cd5eaaf40e1a4947881ef3c490506cfa 2023-04-03 05:54:17,031 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/4d56ed6499e243e38a0ce3d159a72441 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4d56ed6499e243e38a0ce3d159a72441 2023-04-03 05:54:17,037 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4d56ed6499e243e38a0ce3d159a72441, entries=200, sequenceid=309, filesize=32.3 K 2023-04-03 05:54:17,038 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/40ee4ab0a6954716ae520a4cd68f3878 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/40ee4ab0a6954716ae520a4cd68f3878 2023-04-03 05:54:17,044 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/40ee4ab0a6954716ae520a4cd68f3878, entries=150, sequenceid=309, filesize=11.9 K 2023-04-03 05:54:17,045 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/cd5eaaf40e1a4947881ef3c490506cfa as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cd5eaaf40e1a4947881ef3c490506cfa 2023-04-03 05:54:17,049 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cd5eaaf40e1a4947881ef3c490506cfa, entries=150, sequenceid=309, filesize=11.9 K 2023-04-03 05:54:17,050 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=46.96 KB/48090 for 574b09e98d501def4bb65b12ef4de07a in 504ms, sequenceid=309, compaction requested=true 2023-04-03 05:54:17,050 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:17,050 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:17,050 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:17,050 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:17,050 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:17,050 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:17,052 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85368 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:17,052 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/A is initiating minor compaction (all files) 2023-04-03 05:54:17,052 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/A in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,052 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:17,052 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/b5595f86b1374f70b2e36b00405b6fc5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/b9fe9df10cf04a01981686ada3454325, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4d56ed6499e243e38a0ce3d159a72441] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=83.4 K 2023-04-03 05:54:17,052 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/B is initiating minor compaction (all files) 2023-04-03 05:54:17,052 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/B in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,052 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/a7aa188159074109882955611e9f9717, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0f679bef98a746de8cc701df679af2f8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/40ee4ab0a6954716ae520a4cd68f3878] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=35.9 K 2023-04-03 05:54:17,053 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting b5595f86b1374f70b2e36b00405b6fc5, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680501255057 2023-04-03 05:54:17,053 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a7aa188159074109882955611e9f9717, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680501255057 2023-04-03 05:54:17,053 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting b9fe9df10cf04a01981686ada3454325, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680501255380 2023-04-03 05:54:17,053 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0f679bef98a746de8cc701df679af2f8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680501255380 2023-04-03 05:54:17,053 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 4d56ed6499e243e38a0ce3d159a72441, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1680501256542 2023-04-03 05:54:17,054 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 40ee4ab0a6954716ae520a4cd68f3878, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1680501256543 2023-04-03 05:54:17,062 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#B#compaction#151 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:17,070 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:17,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:17,073 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#A#compaction#150 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:17,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:17,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:17,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:17,075 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:17,075 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,075 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:17,075 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:17,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,078 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,079 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,079 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,079 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:17,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:17,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:17,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:17,080 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:17,080 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:17,080 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,080 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:17,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:17,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:17,081 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,081 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:17,081 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:54:17,082 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:17,082 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:17,082 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:17,082 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:17,082 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:17,082 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:17,086 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/09ce6768c0d744329a39744a28c16837 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/09ce6768c0d744329a39744a28c16837 2023-04-03 05:54:17,094 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/B of 574b09e98d501def4bb65b12ef4de07a into 09ce6768c0d744329a39744a28c16837(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:17,095 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:17,095 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/B, priority=13, startTime=1680501257050; duration=0sec 2023-04-03 05:54:17,095 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:17,095 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:17,096 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:17,096 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/C is initiating minor compaction (all files) 2023-04-03 05:54:17,097 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/C in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,097 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/76c33087cd424a3f88e7f87642ea6a91, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/84f688ead23e47ae823c3eb637ec9932, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cd5eaaf40e1a4947881ef3c490506cfa] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=35.9 K 2023-04-03 05:54:17,097 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 76c33087cd424a3f88e7f87642ea6a91, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680501255057 2023-04-03 05:54:17,098 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 84f688ead23e47ae823c3eb637ec9932, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680501255380 2023-04-03 05:54:17,098 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting cd5eaaf40e1a4947881ef3c490506cfa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1680501256543 2023-04-03 05:54:17,101 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403a2b0c4409970424bbf1e97f0c865ef09 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a2b0c4409970424bbf1e97f0c865ef09 2023-04-03 05:54:17,115 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=319, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/925a044853ca4661944904569f2a4ae4 2023-04-03 05:54:17,116 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#C#compaction#153 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:17,135 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/915b354bd1264c65b21f48dc851640db 2023-04-03 05:54:17,156 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/b9ce69f1ca6049ce80ed191035b39c7b 2023-04-03 05:54:17,161 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/925a044853ca4661944904569f2a4ae4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/925a044853ca4661944904569f2a4ae4 2023-04-03 05:54:17,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:17,167 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:17,167 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/925a044853ca4661944904569f2a4ae4, entries=150, sequenceid=319, filesize=25.4 K 2023-04-03 05:54:17,171 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/915b354bd1264c65b21f48dc851640db as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/915b354bd1264c65b21f48dc851640db 2023-04-03 05:54:17,176 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/915b354bd1264c65b21f48dc851640db, entries=150, sequenceid=319, filesize=11.9 K 2023-04-03 05:54:17,177 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/b9ce69f1ca6049ce80ed191035b39c7b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/b9ce69f1ca6049ce80ed191035b39c7b 2023-04-03 05:54:17,181 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/b9ce69f1ca6049ce80ed191035b39c7b, entries=150, sequenceid=319, filesize=11.9 K 2023-04-03 05:54:17,182 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=13.42 KB/13740 for 574b09e98d501def4bb65b12ef4de07a in 101ms, sequenceid=319, compaction requested=false 2023-04-03 05:54:17,182 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:17,182 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:17,182 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:17,183 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,184 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:17,184 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:17,185 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,185 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,186 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,186 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:17,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:17,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c6f0463[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:17,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:17,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,190 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,191 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:17,191 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:17,191 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:17,191 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,191 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:17,192 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:17,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:17,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:17,196 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:17,196 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:17,196 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:17,196 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:17,196 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:17,199 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,199 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,199 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,199 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,201 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,204 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,204 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,204 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,204 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,204 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,204 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:17,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:17,204 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,204 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,204 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,204 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:17,208 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:17,208 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:17,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:17,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:54:17,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:17,240 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:17,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:17,240 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:17,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:17,240 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:17,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501257265, max timestamp=1680501257270Num uniques 859; 2023-04-03 05:54:17,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501257265, max timestamp=1680501257270Num uniques 859; 2023-04-03 05:54:17,272 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:54:17,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501257265, max timestamp=1680501257270Num uniques 859; 2023-04-03 05:54:17,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:54:17,273 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:54:17,294 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501317292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501317293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501317293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501317294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,308 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:17,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:17,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:17,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:17,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:17,311 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:17,311 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,311 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:17,311 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:17,312 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,312 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,313 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,313 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,314 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,314 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:17,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:17,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:17,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:17,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:17,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:17,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,316 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:17,316 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:17,316 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,316 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:17,316 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:17,316 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,316 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:17,316 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:17,318 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,318 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,318 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,318 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:17,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:17,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ecbb703[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:17,320 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:17,320 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,322 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,322 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:17,322 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:17,322 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:17,322 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,322 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:17,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:17,323 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,324 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:17,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:17,328 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:17,328 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:17,328 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:17,328 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:17,328 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:17,328 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:17,330 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,330 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,330 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,332 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,333 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,335 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:17,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,336 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:17,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,336 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,336 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:17,336 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:17,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501317395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,397 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501317396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501317397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,398 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501317397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501317401, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,404 DEBUG [Thread-917] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:54:17,436 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:17,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:17,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:17,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:17,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:17,440 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:17,440 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,440 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:17,440 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:17,440 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,440 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,443 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,443 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,443 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:17,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:17,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:17,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:17,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:17,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:17,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:17,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:17,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:17,445 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,445 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:17,445 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:17,445 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:17,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:17,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:17,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:17,446 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,446 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,447 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,447 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:17,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:17,448 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b4b280a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:17,448 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:17,448 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,449 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,449 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:17,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:17,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:17,450 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,450 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,451 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:17,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:17,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:17,454 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:17,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:17,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:17,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:17,454 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:17,458 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,458 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,458 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,459 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,460 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,462 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:17,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:17,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:17,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,463 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,463 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:17,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,463 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:17,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,530 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/aee339d8d1224ae28eb387bd3cea28a1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/aee339d8d1224ae28eb387bd3cea28a1 2023-04-03 05:54:17,536 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/C of 574b09e98d501def4bb65b12ef4de07a into aee339d8d1224ae28eb387bd3cea28a1(size=12.2 K), total size for store is 24.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:17,536 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:17,536 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/C, priority=13, startTime=1680501257050; duration=0sec 2023-04-03 05:54:17,536 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:17,563 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:17,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:17,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:17,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:17,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:17,566 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:17,566 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:17,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:17,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,568 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,568 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,569 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,569 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:17,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:17,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:17,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:17,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:17,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:17,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:17,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:17,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:17,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:17,571 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:17,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:17,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:17,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:17,573 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:17,575 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:17,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47a1b0b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:17,575 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:17,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:17,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:17,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:17,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,577 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:17,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,577 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,578 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:17,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:17,580 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:17,580 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:17,580 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:17,580 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:17,580 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:17,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,583 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:17,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:17,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,587 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:17,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,587 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:17,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501317598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,599 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501317599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501317599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501317600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,667 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040302c010e4698d49a98ca5c7faa97b2a89 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040302c010e4698d49a98ca5c7faa97b2a89 2023-04-03 05:54:17,673 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=330, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/533b23c44f2f46cfa31ae5f0b176f9b2 2023-04-03 05:54:17,688 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:17,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:17,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:17,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:17,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:17,691 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:17,691 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,691 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:17,691 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:17,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,693 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:17,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:17,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:17,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:17,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:17,695 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,695 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:17,695 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:17,695 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:17,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:17,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:17,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:17,696 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:17,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30824bfb[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:17,699 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:17,699 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,700 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,700 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:17,700 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:17,700 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:17,700 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,700 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:17,702 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,702 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:17,702 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,702 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:17,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:17,704 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:17,704 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:17,704 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:17,704 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:17,704 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:17,706 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,706 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,706 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,706 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,708 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,710 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:17,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:17,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,711 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:17,711 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:17,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,811 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:17,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:17,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:17,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:17,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:17,814 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:17,814 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,814 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:17,814 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:17,815 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,815 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,816 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,816 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,816 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,816 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:17,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:17,817 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:17,817 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:17,817 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:17,817 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,817 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:17,818 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:17,818 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:17,818 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,818 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:17,818 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:17,818 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,818 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:17,818 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:17,818 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,818 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:17,818 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:17,821 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,821 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,821 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,821 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:17,822 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:17,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13d28147[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:17,822 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,823 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:17,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:17,824 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,824 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:17,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:17,825 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,825 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:17,825 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,825 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:17,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:17,829 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:17,829 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:17,829 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:17,829 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:17,829 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:17,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,831 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,841 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,841 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,841 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,842 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:17,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:17,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,842 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:17,842 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:17,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,900 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/35e404745f5d4eaf9c4eb8ef906e7f40 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/35e404745f5d4eaf9c4eb8ef906e7f40 2023-04-03 05:54:17,904 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/A of 574b09e98d501def4bb65b12ef4de07a into 35e404745f5d4eaf9c4eb8ef906e7f40(size=25.8 K), total size for store is 51.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:17,905 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:17,905 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/A, priority=13, startTime=1680501257050; duration=0sec 2023-04-03 05:54:17,905 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:17,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501317900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501317901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501317904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:17,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501317905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,942 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:17,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:17,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:17,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:17,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:17,946 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:17,946 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,946 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:17,946 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:17,946 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,946 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,947 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,947 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:17,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:17,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:17,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:17,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:17,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:17,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:17,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:17,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:17,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:17,950 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,950 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:17,950 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:17,950 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:17,950 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:17,950 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:17,950 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,950 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:17,950 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:17,951 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,951 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,951 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:17,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:17,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@519be04f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:17,953 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:17,953 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,955 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:17,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:17,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:17,955 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,955 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:17,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:17,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:17,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:17,959 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:17,960 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:17,960 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:17,960 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:17,960 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:17,961 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,961 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,961 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:17,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:17,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:17,962 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,963 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:17,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:17,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:17,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:17,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:17,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:17,966 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:17,966 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:17,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:17,966 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:17,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:17,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:17,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:17,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:17,966 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:17,966 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:18,066 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:18,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:18,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:18,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:18,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:18,070 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:18,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:18,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:18,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:18,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,073 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:18,073 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:18,073 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,073 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:18,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:18,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:18,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:18,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:18,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:18,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:18,077 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,077 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:18,077 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:18,077 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:18,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:18,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:18,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:18,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:18,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ed1b34[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:18,081 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:18,081 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,082 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,082 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:18,082 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,082 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:18,082 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:18,082 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:18,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:18,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,086 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/8b43f88f49674993bfa722c1240f676c 2023-04-03 05:54:18,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:18,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:18,089 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:18,089 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:18,089 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:18,089 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:18,089 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:18,093 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,093 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,093 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:18,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:18,093 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:18,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,095 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:18,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:18,098 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:18,099 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:18,099 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:18,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,099 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:18,099 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:18,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:18,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:18,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:18,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:18,100 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/44aa33264db04119807a4ee52108a4bd 2023-04-03 05:54:18,105 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/533b23c44f2f46cfa31ae5f0b176f9b2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/533b23c44f2f46cfa31ae5f0b176f9b2 2023-04-03 05:54:18,110 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/533b23c44f2f46cfa31ae5f0b176f9b2, entries=350, sequenceid=330, filesize=52.9 K 2023-04-03 05:54:18,111 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/8b43f88f49674993bfa722c1240f676c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/8b43f88f49674993bfa722c1240f676c 2023-04-03 05:54:18,115 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/8b43f88f49674993bfa722c1240f676c, entries=150, sequenceid=330, filesize=11.9 K 2023-04-03 05:54:18,116 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/44aa33264db04119807a4ee52108a4bd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/44aa33264db04119807a4ee52108a4bd 2023-04-03 05:54:18,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/44aa33264db04119807a4ee52108a4bd, entries=150, sequenceid=330, filesize=11.9 K 2023-04-03 05:54:18,121 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 574b09e98d501def4bb65b12ef4de07a in 882ms, sequenceid=330, compaction requested=true 2023-04-03 05:54:18,121 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:18,121 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:18,121 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:18,121 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:18,121 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:18,121 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:18,122 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:18,122 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:18,122 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/A is initiating minor compaction (all files) 2023-04-03 05:54:18,122 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/B is initiating minor compaction (all files) 2023-04-03 05:54:18,122 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/A in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,122 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/B in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,122 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/35e404745f5d4eaf9c4eb8ef906e7f40, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/925a044853ca4661944904569f2a4ae4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/533b23c44f2f46cfa31ae5f0b176f9b2] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=104.1 K 2023-04-03 05:54:18,122 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/09ce6768c0d744329a39744a28c16837, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/915b354bd1264c65b21f48dc851640db, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/8b43f88f49674993bfa722c1240f676c] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=36.0 K 2023-04-03 05:54:18,123 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 35e404745f5d4eaf9c4eb8ef906e7f40, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1680501256543 2023-04-03 05:54:18,123 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 09ce6768c0d744329a39744a28c16837, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1680501256543 2023-04-03 05:54:18,123 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 925a044853ca4661944904569f2a4ae4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680501256549 2023-04-03 05:54:18,123 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 915b354bd1264c65b21f48dc851640db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680501256549 2023-04-03 05:54:18,124 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 533b23c44f2f46cfa31ae5f0b176f9b2, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680501257166 2023-04-03 05:54:18,124 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 8b43f88f49674993bfa722c1240f676c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680501257168 2023-04-03 05:54:18,133 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#B#compaction#160 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:18,136 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#A#compaction#159 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:18,158 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/7e634fb8d35e4be391e67b6c26e26033 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/7e634fb8d35e4be391e67b6c26e26033 2023-04-03 05:54:18,163 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/B of 574b09e98d501def4bb65b12ef4de07a into 7e634fb8d35e4be391e67b6c26e26033(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:18,163 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:18,163 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/B, priority=13, startTime=1680501258121; duration=0sec 2023-04-03 05:54:18,163 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:18,163 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:18,164 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:18,165 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/C is initiating minor compaction (all files) 2023-04-03 05:54:18,165 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/C in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,165 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/aee339d8d1224ae28eb387bd3cea28a1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/b9ce69f1ca6049ce80ed191035b39c7b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/44aa33264db04119807a4ee52108a4bd] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=36.0 K 2023-04-03 05:54:18,165 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting aee339d8d1224ae28eb387bd3cea28a1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=309, earliestPutTs=1680501256543 2023-04-03 05:54:18,166 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b9ce69f1ca6049ce80ed191035b39c7b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=319, earliestPutTs=1680501256549 2023-04-03 05:54:18,166 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 44aa33264db04119807a4ee52108a4bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680501257168 2023-04-03 05:54:18,172 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#C#compaction#161 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:18,182 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/1946cd0708044022a3f0309e631f89f0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/1946cd0708044022a3f0309e631f89f0 2023-04-03 05:54:18,188 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/C of 574b09e98d501def4bb65b12ef4de07a into 1946cd0708044022a3f0309e631f89f0(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:18,188 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:18,188 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/C, priority=13, startTime=1680501258121; duration=0sec 2023-04-03 05:54:18,188 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:18,199 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:18,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:18,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:18,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:18,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:18,202 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:18,202 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:18,202 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:18,203 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:18,203 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,203 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,204 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:18,204 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:18,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:18,205 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,205 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,205 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:18,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:18,206 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:18,206 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:18,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:18,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:18,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:18,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:18,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:18,206 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,206 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:18,206 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-03 05:54:18,206 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:18,207 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:18,207 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:18,207 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:18,207 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:18,207 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:18,234 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403be81890d15664f29a46a6e935dc91b47 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403be81890d15664f29a46a6e935dc91b47 2023-04-03 05:54:18,256 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=370, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/12272198cc8740bb9b409d9147b8656e 2023-04-03 05:54:18,277 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/0337a461fb8b4b69b26e911f59989be0 2023-04-03 05:54:18,289 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=370 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/700e20937e2242b3a1a0dac184b33898 2023-04-03 05:54:18,295 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/12272198cc8740bb9b409d9147b8656e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/12272198cc8740bb9b409d9147b8656e 2023-04-03 05:54:18,308 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/12272198cc8740bb9b409d9147b8656e, entries=150, sequenceid=370, filesize=25.4 K 2023-04-03 05:54:18,309 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/0337a461fb8b4b69b26e911f59989be0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0337a461fb8b4b69b26e911f59989be0 2023-04-03 05:54:18,320 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0337a461fb8b4b69b26e911f59989be0, entries=150, sequenceid=370, filesize=11.9 K 2023-04-03 05:54:18,321 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/700e20937e2242b3a1a0dac184b33898 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/700e20937e2242b3a1a0dac184b33898 2023-04-03 05:54:18,336 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/700e20937e2242b3a1a0dac184b33898, entries=150, sequenceid=370, filesize=11.9 K 2023-04-03 05:54:18,337 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=0 B/0 for 574b09e98d501def4bb65b12ef4de07a in 131ms, sequenceid=370, compaction requested=false 2023-04-03 05:54:18,337 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:18,337 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,337 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:18,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:18,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:18,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:18,339 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,339 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:18,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@279255e1[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:18,342 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:18,342 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,344 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,344 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:18,344 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:18,344 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,344 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:18,344 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:18,345 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:18,345 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:18,345 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:18,350 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:18,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:18,353 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:18,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:18,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:18,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:18,354 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:18,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,356 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:18,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:18,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,357 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,364 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,364 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:18,364 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:18,364 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,364 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:18,364 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:18,364 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,364 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,364 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,364 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:18,364 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:18,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:18,364 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:18,365 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:18,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:18,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:18,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:18,442 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:54:18,443 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:18,443 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:18,443 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:18,443 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:18,443 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:18,443 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:18,464 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304038fd8ce82e5e54013bf86b06b4a735c41 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038fd8ce82e5e54013bf86b06b4a735c41 2023-04-03 05:54:18,465 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:18,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:18,468 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:18,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:18,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:18,469 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:18,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:18,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:18,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:18,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,472 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=381, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/ec0fa778724645bdbe09b20545f48eea 2023-04-03 05:54:18,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:18,472 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:18,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:18,473 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,473 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,474 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:18,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:18,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:18,475 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:18,475 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:18,476 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,476 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:18,476 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:18,476 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:18,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:18,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:18,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:18,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:18,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:18,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@204d20ed[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:18,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:18,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:18,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:18,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:18,484 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,484 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:18,486 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:18,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:18,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:18,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:18,490 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:18,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:18,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:18,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:18,490 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:18,491 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:18,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680501258445, max timestamp=1680501258489Num uniques 859; 2023-04-03 05:54:18,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501318485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,491 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-03 05:54:18,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680501258445, max timestamp=1680501258489Num uniques 859; 2023-04-03 05:54:18,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-03 05:54:18,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=859, cellSize=39335, totalHeapSize=89576, min timestamp=1680501258445, max timestamp=1680501258489Num uniques 859; 2023-04-03 05:54:18,492 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=39335, suffix heap size=117064,new segment heap size=89576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=859 2023-04-03 05:54:18,492 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,492 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,493 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:18,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,493 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,496 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,500 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:18,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:18,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:18,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:18,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:18,501 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:18,501 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,501 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,501 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,501 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:18,501 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:18,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:18,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:18,503 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/fa0cc3f9d23146789a9ad27a5c581f95 2023-04-03 05:54:18,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:18,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501318500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,505 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:18,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501318500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,505 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:18,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501318501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,594 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:18,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501318593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,601 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:18,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:18,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:18,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:18,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:18,604 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:18,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:18,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:18,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:18,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,605 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,607 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:18,607 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:18,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:18,607 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,607 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:18,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:18,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:18,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:18,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:18,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:18,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:18,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:18,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:18,608 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,608 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:18,609 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:18,609 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,609 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:18,609 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:18,609 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,609 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:18,609 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:18,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:18,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501318606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:18,610 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501318606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,610 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:18,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501318607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,611 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,611 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:18,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:18,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2eda3533[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:18,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:18,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,613 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,613 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:18,613 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:18,613 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,614 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:18,614 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:18,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,615 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,615 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:18,615 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:18,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:18,617 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:18,617 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:18,617 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:18,617 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:18,617 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:18,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:18,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:18,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,621 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,624 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,624 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,624 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,624 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:18,624 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:18,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:18,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:18,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:18,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:18,624 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:18,624 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:18,624 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,624 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:18,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:18,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:18,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:18,624 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,624 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,624 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,724 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:18,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:18,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:18,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:18,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:18,728 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:18,728 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:18,728 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:18,729 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:18,729 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,729 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,730 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:18,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:18,730 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,732 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,732 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:18,732 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:18,733 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:18,733 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:18,733 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,733 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:18,734 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:18,734 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,734 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:18,734 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:18,734 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,734 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:18,734 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,735 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,735 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:18,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:18,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4383af67[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:18,737 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:18,737 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,739 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:18,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:18,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:18,740 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,740 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:18,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:18,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:18,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:18,741 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:18,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:18,752 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:18,752 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:18,752 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:18,752 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:18,752 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:18,754 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,754 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,754 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:18,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:18,756 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,761 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:18,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:18,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:18,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:18,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:18,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:18,766 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:18,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:18,766 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:18,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,767 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,767 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:18,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:18,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501318797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:18,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501318811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:18,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53226 deadline: 1680501318811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:18,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501318812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,867 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:18,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:18,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:18,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:18,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:18,873 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:18,873 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:18,873 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:18,873 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:18,873 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,873 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:18,875 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:18,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:18,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:18,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:18,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:18,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:18,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:18,877 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:18,877 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:18,877 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:18,878 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:18,878 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:18,878 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,878 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:18,878 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:18,878 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,878 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:18,878 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:18,878 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,878 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:18,878 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:18,883 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,883 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:18,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768bd7ab[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:18,899 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:18,899 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,900 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,900 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:18,900 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:18,900 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:18,901 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,901 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:18,902 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:18,902 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:18,902 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:18,902 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:18,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:18,912 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:18,912 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:18,912 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:18,913 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:18,913 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:18,915 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,915 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:18,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:18,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,916 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:18,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:18,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/ac11d0f368c74c50b8197dc5dc443404 2023-04-03 05:54:18,923 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,924 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/ec0fa778724645bdbe09b20545f48eea as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/ec0fa778724645bdbe09b20545f48eea 2023-04-03 05:54:18,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:18,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:18,928 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/ec0fa778724645bdbe09b20545f48eea, entries=400, sequenceid=381, filesize=59.8 K 2023-04-03 05:54:18,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:18,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,931 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/fa0cc3f9d23146789a9ad27a5c581f95 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/fa0cc3f9d23146789a9ad27a5c581f95 2023-04-03 05:54:18,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:18,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:18,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:18,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:18,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:18,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:18,934 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,934 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,934 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:18,934 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:18,934 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,934 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:18,934 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:18,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:18,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:18,934 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:18,935 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:18,935 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:18,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/fa0cc3f9d23146789a9ad27a5c581f95, entries=400, sequenceid=381, filesize=23.8 K 2023-04-03 05:54:18,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/ac11d0f368c74c50b8197dc5dc443404 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/ac11d0f368c74c50b8197dc5dc443404 2023-04-03 05:54:18,941 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/ac11d0f368c74c50b8197dc5dc443404, entries=150, sequenceid=381, filesize=11.9 K 2023-04-03 05:54:18,941 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-03 05:54:18,942 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=147.60 KB/151140 for 574b09e98d501def4bb65b12ef4de07a in 500ms, sequenceid=381, compaction requested=true 2023-04-03 05:54:18,942 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:18,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:18,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:18,942 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 05:54:18,942 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-03 05:54:18,943 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 05:54:18,943 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 05:54:18,943 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. because compaction request was cancelled 2023-04-03 05:54:18,943 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:18,944 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 49206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:18,944 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/B is initiating minor compaction (all files) 2023-04-03 05:54:18,944 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/B in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,944 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/7e634fb8d35e4be391e67b6c26e26033, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0337a461fb8b4b69b26e911f59989be0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/fa0cc3f9d23146789a9ad27a5c581f95] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=48.1 K 2023-04-03 05:54:18,944 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 7e634fb8d35e4be391e67b6c26e26033, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680501257168 2023-04-03 05:54:18,945 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0337a461fb8b4b69b26e911f59989be0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680501257289 2023-04-03 05:54:18,946 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting fa0cc3f9d23146789a9ad27a5c581f95, keycount=400, bloomtype=ROW, size=23.8 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680501258423 2023-04-03 05:54:18,953 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#B#compaction#168 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:18,967 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/c38dea87be0448f99faa9cc02ed14aff as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/c38dea87be0448f99faa9cc02ed14aff 2023-04-03 05:54:18,975 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/B of 574b09e98d501def4bb65b12ef4de07a into c38dea87be0448f99faa9cc02ed14aff(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:18,975 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:18,975 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/B, priority=13, startTime=1680501258942; duration=0sec 2023-04-03 05:54:18,975 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:18,975 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:18,978 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:18,978 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/C is initiating minor compaction (all files) 2023-04-03 05:54:18,978 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/C in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:18,978 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/1946cd0708044022a3f0309e631f89f0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/700e20937e2242b3a1a0dac184b33898, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/ac11d0f368c74c50b8197dc5dc443404] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=36.1 K 2023-04-03 05:54:18,980 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/2fe03fd0f5e9425bb4e57d70c4a7e00d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/2fe03fd0f5e9425bb4e57d70c4a7e00d 2023-04-03 05:54:18,981 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1946cd0708044022a3f0309e631f89f0, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680501257168 2023-04-03 05:54:18,982 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 700e20937e2242b3a1a0dac184b33898, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680501257289 2023-04-03 05:54:18,983 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ac11d0f368c74c50b8197dc5dc443404, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680501258429 2023-04-03 05:54:18,994 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/A of 574b09e98d501def4bb65b12ef4de07a into 2fe03fd0f5e9425bb4e57d70c4a7e00d(size=25.9 K), total size for store is 111.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:18,994 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:18,994 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/A, priority=13, startTime=1680501258121; duration=0sec 2023-04-03 05:54:18,994 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:18,998 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#C#compaction#169 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:19,035 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:19,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:19,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:19,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:19,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:19,038 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:19,038 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:19,038 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:19,038 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:19,039 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:19,039 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:19,040 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,040 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:19,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:19,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:19,040 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,041 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:19,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:19,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:19,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:19,042 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:19,042 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:19,042 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:19,042 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:19,042 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:19,049 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:19,049 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:19,049 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:19,050 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:19,050 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=147.60 KB heapSize=359.20 KB 2023-04-03 05:54:19,050 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:19,050 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:19,050 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:19,050 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:19,050 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:19,050 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:19,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:19,102 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:19,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501319115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501319115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501319116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501319117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501319219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501319219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501319220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,412 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/2812a5fff78049fe87168a467d5ae6b9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/2812a5fff78049fe87168a467d5ae6b9 2023-04-03 05:54:19,417 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/C of 574b09e98d501def4bb65b12ef4de07a into 2812a5fff78049fe87168a467d5ae6b9(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:19,417 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:19,417 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/C, priority=13, startTime=1680501258942; duration=0sec 2023-04-03 05:54:19,417 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:19,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53196 deadline: 1680501319422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501319423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501319423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,468 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304038621fd5519a94e4a99e15b30adac10a4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038621fd5519a94e4a99e15b30adac10a4 2023-04-03 05:54:19,475 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=408, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/40182c28094249e49e26a7b020d2cb5b 2023-04-03 05:54:19,620 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501319619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,724 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501319724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501319724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:19,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501319726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,887 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/395118226dca4c2b97385cdd755eac12 2023-04-03 05:54:19,906 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=408 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/94194a4eb38f4c58bf9335f235e5e64d 2023-04-03 05:54:19,913 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/40182c28094249e49e26a7b020d2cb5b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/40182c28094249e49e26a7b020d2cb5b 2023-04-03 05:54:19,919 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/40182c28094249e49e26a7b020d2cb5b, entries=150, sequenceid=408, filesize=25.4 K 2023-04-03 05:54:19,920 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/395118226dca4c2b97385cdd755eac12 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/395118226dca4c2b97385cdd755eac12 2023-04-03 05:54:19,926 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/395118226dca4c2b97385cdd755eac12, entries=150, sequenceid=408, filesize=11.9 K 2023-04-03 05:54:19,927 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/94194a4eb38f4c58bf9335f235e5e64d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/94194a4eb38f4c58bf9335f235e5e64d 2023-04-03 05:54:19,946 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/94194a4eb38f4c58bf9335f235e5e64d, entries=150, sequenceid=408, filesize=11.9 K 2023-04-03 05:54:19,947 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~359.16 KB/367776, currentSize=53.67 KB/54960 for 574b09e98d501def4bb65b12ef4de07a in 897ms, sequenceid=408, compaction requested=true 2023-04-03 05:54:19,947 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:19,947 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:19,947 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:19,948 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:19,950 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:19,950 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:19,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:19,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:19,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:19,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f08cbdf[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:19,952 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:19,952 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:19,955 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:19,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:19,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:19,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:19,955 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,955 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:19,956 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:19,956 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:19,956 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:19,960 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:19,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:19,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:19,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:19,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:19,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:19,964 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:19,964 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:19,964 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:19,964 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:19,964 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:19,966 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:19,966 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:19,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:19,966 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:19,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:19,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:19,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:19,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:19,969 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,972 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:19,972 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,972 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:19,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:19,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:19,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:19,972 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:19,972 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:19,972 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:19,973 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:19,973 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:19,973 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:19,973 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:19,973 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:19,972 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:19,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:19,973 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:20,073 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:20,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:20,076 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:20,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:20,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:20,077 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:20,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:20,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:20,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:20,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,088 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,088 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:20,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:20,089 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,089 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:20,089 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:20,090 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:20,090 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:20,090 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:20,090 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:20,091 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:20,091 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:20,091 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:20,091 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:20,091 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 05:54:20,091 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:20,091 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:20,091 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:20,091 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:20,091 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:20,091 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:20,156 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403a1252c7d497246029ccb0e66e984d8a8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a1252c7d497246029ccb0e66e984d8a8 2023-04-03 05:54:20,179 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=420, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/63c5685a1e2e4d469be85bc9833f8280 2023-04-03 05:54:20,207 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/ae38866e270443cea646fe8fe8afe911 2023-04-03 05:54:20,224 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=420 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/b70ff2c1da6d45118384da6e822e63e4 2023-04-03 05:54:20,231 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/63c5685a1e2e4d469be85bc9833f8280 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/63c5685a1e2e4d469be85bc9833f8280 2023-04-03 05:54:20,234 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:20,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:20,237 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/63c5685a1e2e4d469be85bc9833f8280, entries=100, sequenceid=420, filesize=18.6 K 2023-04-03 05:54:20,238 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/ae38866e270443cea646fe8fe8afe911 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/ae38866e270443cea646fe8fe8afe911 2023-04-03 05:54:20,244 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/ae38866e270443cea646fe8fe8afe911, entries=100, sequenceid=420, filesize=9.5 K 2023-04-03 05:54:20,245 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/b70ff2c1da6d45118384da6e822e63e4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/b70ff2c1da6d45118384da6e822e63e4 2023-04-03 05:54:20,256 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/b70ff2c1da6d45118384da6e822e63e4, entries=100, sequenceid=420, filesize=9.5 K 2023-04-03 05:54:20,257 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=33.54 KB/34350 for 574b09e98d501def4bb65b12ef4de07a in 166ms, sequenceid=420, compaction requested=true 2023-04-03 05:54:20,257 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:20,258 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:20,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:20,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:20,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:20,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:20,259 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,259 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:20,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:20,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@400412a0[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:20,263 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:20,264 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,265 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,265 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:20,268 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,268 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:20,268 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:20,268 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:20,270 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,270 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:20,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:20,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:20,273 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:20,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:20,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:20,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:20,274 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:20,275 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:54:20,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:20,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:20,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:20,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:20,275 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:20,275 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:20,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:20,276 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,276 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,276 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,278 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,282 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,284 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:20,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:20,285 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:20,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:20,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:20,286 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:20,286 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:20,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:20,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:20,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:20,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:20,333 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403963f7380baec42d6bce3a1af72725558 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403963f7380baec42d6bce3a1af72725558 2023-04-03 05:54:20,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501260322, max timestamp=1680501260337Num uniques 859; 2023-04-03 05:54:20,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501260322, max timestamp=1680501260337Num uniques 859; 2023-04-03 05:54:20,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:54:20,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501260322, max timestamp=1680501260337Num uniques 859; 2023-04-03 05:54:20,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:54:20,339 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:54:20,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:20,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501320369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:20,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501320373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:20,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501320374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,386 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:20,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:20,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:20,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:20,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:20,391 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:20,391 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:20,391 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:20,392 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:20,392 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,392 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,393 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:20,393 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:20,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:20,394 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,394 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:20,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:20,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:20,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:20,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:20,395 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:20,395 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:20,395 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:20,395 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:20,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:20,396 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,396 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:20,396 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:20,397 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,398 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,398 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:20,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f93bc8b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:20,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:20,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,410 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,410 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:20,411 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:20,411 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:20,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:20,417 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:20,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:20,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:20,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:20,431 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:20,431 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:20,431 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:20,431 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:20,432 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:20,435 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,435 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,435 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:20,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:20,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:20,436 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,441 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,446 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,446 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,446 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,446 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,446 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:20,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:20,446 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:20,446 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,446 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:20,447 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:20,447 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:20,447 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:20,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:20,447 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:20,447 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:20,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:20,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501320478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:20,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501320478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:20,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501320478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,547 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:20,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:20,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:20,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:20,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:20,550 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:20,550 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:20,550 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:20,550 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:20,551 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,551 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,552 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:20,552 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:20,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,552 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,553 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:20,553 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,553 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:20,553 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:20,553 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:20,554 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:20,554 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:20,554 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:20,554 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:20,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:20,556 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,557 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,557 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,557 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:20,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f6236cb[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:20,558 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:20,559 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,560 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,560 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:20,560 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:20,560 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:20,560 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,560 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:20,563 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:20,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:20,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:20,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:20,571 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:20,571 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:20,571 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:20,571 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:20,571 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:20,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,573 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,573 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,573 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,574 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:20,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:20,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:20,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:20,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:20,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:20,577 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:20,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:20,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:20,577 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:20,577 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:20,631 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:20,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501320630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,677 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:20,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:20,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:20,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:20,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:20,682 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:20,682 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:20,682 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:20,682 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:20,682 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,682 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:20,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501320683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,684 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:20,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:20,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:20,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:20,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501320684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,685 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:20,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,686 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:20,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:20,686 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:20,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:20,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:20,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:20,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:20,711 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:20,711 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:20,711 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:20,711 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:20,711 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:20,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:20,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:20,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501320711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,713 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,713 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:20,715 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,715 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,715 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:20,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:20,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22674e82[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:20,717 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:20,717 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,718 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,718 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:20,718 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:20,718 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:20,718 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,718 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:20,720 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,720 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,720 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:20,720 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:20,722 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:20,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:20,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:20,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:20,722 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:20,724 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,724 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:20,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,724 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:20,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,725 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:20,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:20,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:20,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:20,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:20,728 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:20,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:20,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:20,729 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:20,729 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:20,729 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:20,729 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,729 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:20,729 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,764 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=430, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/02cc96ae79bb4345be0ad8426bd85cfa 2023-04-03 05:54:20,829 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:20,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:20,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:20,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:20,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:20,833 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:20,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:20,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:20,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:20,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,835 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:20,835 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:20,835 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,835 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:20,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:20,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:20,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:20,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:20,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:20,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:20,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:20,837 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:20,837 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:20,837 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:20,837 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:20,838 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,838 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,839 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:20,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72f64421[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:20,840 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:20,840 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,841 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,841 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,842 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:20,842 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:20,842 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:20,842 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:20,844 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:20,844 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,844 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,844 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:20,846 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:20,846 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:20,846 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:20,846 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:20,846 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:20,846 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:20,851 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,851 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,851 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:20,851 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,855 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:20,855 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:20,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:20,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:20,855 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,855 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,856 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,856 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,856 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:20,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:20,856 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,856 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,857 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:20,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:20,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:20,858 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:20,858 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:20,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:20,958 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:20,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:20,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:20,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:20,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:20,966 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:20,966 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:20,966 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:20,966 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:20,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,968 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:20,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:20,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:20,969 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:20,984 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:20,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:20,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:20,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:20,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:20,986 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:20,986 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:20,986 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:20,986 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:20,987 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:20,988 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:20,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:20,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:20,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:20,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:20,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:20,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501320986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:20,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501320987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,991 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:20,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:20,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:20,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2053d910[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:20,995 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:20,995 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,996 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:20,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:20,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:20,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:20,997 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,997 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:20,999 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:20,999 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:20,999 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:20,999 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:21,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:21,003 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:21,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:21,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:21,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:21,003 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:21,005 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,006 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,006 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:21,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,008 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,016 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:21,016 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:21,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:21,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:21,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:21,017 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,017 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,017 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:21,017 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:21,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:21,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,018 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:21,018 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:21,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:21,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501321015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,118 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:21,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:21,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:21,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:21,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:21,121 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:21,121 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:21,121 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:21,121 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:21,122 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,122 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,123 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:21,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:21,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,124 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,124 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:21,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,124 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:21,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:21,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:21,125 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:21,125 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:21,125 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:21,125 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:21,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,127 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:21,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eb977bf[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:21,129 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:21,129 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,130 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,131 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:21,131 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:21,131 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:21,131 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,131 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:21,133 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,133 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:21,133 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:21,133 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:21,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:21,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:21,136 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:21,136 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:21,136 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:21,136 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:21,136 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:21,138 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,138 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,138 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:21,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:21,139 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,140 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,142 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,142 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,142 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,142 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:21,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:21,142 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,142 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,142 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:21,142 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:21,142 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:21,142 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:21,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:21,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:21,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:21,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:21,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:21,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/49edc184a0fb49e0a4a5f2887a5e89fc 2023-04-03 05:54:21,242 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:21,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:21,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:21,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:21,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:21,246 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:21,246 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:21,246 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:21,246 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:21,246 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,246 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,247 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:21,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,248 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,248 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:21,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,249 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:21,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:21,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:21,250 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:21,250 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:21,250 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:21,250 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:21,250 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:21,257 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63229b4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:21,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:21,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,260 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:21,260 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,260 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:21,262 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,262 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:21,262 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,262 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:21,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:21,265 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:21,265 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:21,265 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:21,265 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:21,265 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:21,266 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,266 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,266 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,267 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,268 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,274 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,274 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:21,274 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:21,274 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:21,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:21,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:21,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:21,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:21,274 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,275 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,275 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,275 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,275 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,275 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:21,275 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:21,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:21,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:21,275 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:21,375 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:21,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:21,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:21,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:21,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:21,378 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:21,378 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:21,378 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:21,378 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:21,379 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,379 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,380 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:21,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:21,380 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,380 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,380 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:21,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:21,381 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:21,381 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:21,383 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:21,383 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:21,383 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:21,383 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:21,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,384 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,385 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,385 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4625b21f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:21,386 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:21,386 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,388 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:21,388 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,389 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:21,388 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:21,390 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,390 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:21,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:21,392 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:21,392 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:21,392 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:21,392 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:21,392 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:21,393 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,393 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,393 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,394 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,395 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,397 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:21,397 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,397 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:21,397 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:21,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:21,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:21,397 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:21,398 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:21,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:21,398 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,398 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:21,398 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,398 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:21,398 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,398 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,426 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:21,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501321426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,428 DEBUG [Thread-917] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8173 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:54:21,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:21,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501321491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:21,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501321493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,498 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:21,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:21,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:21,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:21,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:21,501 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:21,501 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:21,502 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:21,502 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:21,502 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,502 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,504 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:21,504 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:21,504 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,505 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:21,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:21,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:21,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:21,506 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:21,506 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:21,506 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:21,506 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:21,506 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,507 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:21,507 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,508 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,508 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:21,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78806892[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:21,510 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:21,510 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,511 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:21,511 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:21,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:21,511 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:21,513 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:21,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:21,515 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:21,515 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:21,515 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:21,515 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:21,515 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:21,518 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,518 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:21,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,518 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:21,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:21,518 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:21,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,519 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:21,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,521 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:21,521 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:21,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:21,521 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:21,522 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:21,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:21,522 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,522 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:21,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:21,522 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:21,522 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,522 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:21,522 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:21,522 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,526 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:21,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501321524, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,587 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/cea57392e93e4081ab12f4167a6c0d11 2023-04-03 05:54:21,591 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/02cc96ae79bb4345be0ad8426bd85cfa as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/02cc96ae79bb4345be0ad8426bd85cfa 2023-04-03 05:54:21,595 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/02cc96ae79bb4345be0ad8426bd85cfa, entries=350, sequenceid=430, filesize=52.9 K 2023-04-03 05:54:21,596 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/49edc184a0fb49e0a4a5f2887a5e89fc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/49edc184a0fb49e0a4a5f2887a5e89fc 2023-04-03 05:54:21,600 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/49edc184a0fb49e0a4a5f2887a5e89fc, entries=150, sequenceid=430, filesize=11.9 K 2023-04-03 05:54:21,600 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/cea57392e93e4081ab12f4167a6c0d11 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cea57392e93e4081ab12f4167a6c0d11 2023-04-03 05:54:21,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cea57392e93e4081ab12f4167a6c0d11, entries=150, sequenceid=430, filesize=11.9 K 2023-04-03 05:54:21,607 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 574b09e98d501def4bb65b12ef4de07a in 1332ms, sequenceid=430, compaction requested=true 2023-04-03 05:54:21,607 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:21,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:21,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:21,607 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 05:54:21,607 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:21,607 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:21,608 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46846 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:21,608 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/B is initiating minor compaction (all files) 2023-04-03 05:54:21,608 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/B in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:21,609 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/c38dea87be0448f99faa9cc02ed14aff, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/395118226dca4c2b97385cdd755eac12, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/ae38866e270443cea646fe8fe8afe911, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/49edc184a0fb49e0a4a5f2887a5e89fc] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=45.7 K 2023-04-03 05:54:21,609 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 213048 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 05:54:21,609 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/A is initiating minor compaction (all files) 2023-04-03 05:54:21,609 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/A in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:21,609 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting c38dea87be0448f99faa9cc02ed14aff, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680501258429 2023-04-03 05:54:21,609 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/2fe03fd0f5e9425bb4e57d70c4a7e00d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/12272198cc8740bb9b409d9147b8656e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/ec0fa778724645bdbe09b20545f48eea, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/40182c28094249e49e26a7b020d2cb5b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/63c5685a1e2e4d469be85bc9833f8280, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/02cc96ae79bb4345be0ad8426bd85cfa] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=208.1 K 2023-04-03 05:54:21,609 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 395118226dca4c2b97385cdd755eac12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680501258489 2023-04-03 05:54:21,609 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 2fe03fd0f5e9425bb4e57d70c4a7e00d, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=330, earliestPutTs=1680501257168 2023-04-03 05:54:21,610 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ae38866e270443cea646fe8fe8afe911, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680501259114 2023-04-03 05:54:21,610 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 12272198cc8740bb9b409d9147b8656e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=370, earliestPutTs=1680501257289 2023-04-03 05:54:21,610 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 49edc184a0fb49e0a4a5f2887a5e89fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680501260251 2023-04-03 05:54:21,610 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting ec0fa778724645bdbe09b20545f48eea, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680501258423 2023-04-03 05:54:21,610 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 40182c28094249e49e26a7b020d2cb5b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680501258489 2023-04-03 05:54:21,611 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 63c5685a1e2e4d469be85bc9833f8280, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680501259114 2023-04-03 05:54:21,611 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 02cc96ae79bb4345be0ad8426bd85cfa, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680501260233 2023-04-03 05:54:21,617 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#B#compaction#179 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:21,622 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:21,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:21,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:21,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:21,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:21,625 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:21,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:21,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:21,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:21,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,627 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:21,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:21,629 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:21,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:21,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:21,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:21,630 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:21,630 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:21,630 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:21,630 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:21,630 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:21,630 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:21,630 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:21,631 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:21,631 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:21,631 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-03 05:54:21,631 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:21,631 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:21,631 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:21,631 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:21,631 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:21,631 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:21,637 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#A#compaction#180 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:21,656 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/ca028b3d0eec450cb2d992fdba2ec433 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/ca028b3d0eec450cb2d992fdba2ec433 2023-04-03 05:54:21,661 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/B of 574b09e98d501def4bb65b12ef4de07a into ca028b3d0eec450cb2d992fdba2ec433(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:21,661 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:21,661 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/B, priority=12, startTime=1680501261607; duration=0sec 2023-04-03 05:54:21,661 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:21,661 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:21,663 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46846 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:21,663 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/C is initiating minor compaction (all files) 2023-04-03 05:54:21,663 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/C in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:21,663 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/2812a5fff78049fe87168a467d5ae6b9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/94194a4eb38f4c58bf9335f235e5e64d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/b70ff2c1da6d45118384da6e822e63e4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cea57392e93e4081ab12f4167a6c0d11] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=45.7 K 2023-04-03 05:54:21,663 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 2812a5fff78049fe87168a467d5ae6b9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1680501258429 2023-04-03 05:54:21,664 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 94194a4eb38f4c58bf9335f235e5e64d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=408, earliestPutTs=1680501258489 2023-04-03 05:54:21,664 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b70ff2c1da6d45118384da6e822e63e4, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=420, earliestPutTs=1680501259114 2023-04-03 05:54:21,665 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting cea57392e93e4081ab12f4167a6c0d11, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680501260251 2023-04-03 05:54:21,681 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#C#compaction#182 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:21,687 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403c4bd8cbc4aa740efb004579574a0e1e2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c4bd8cbc4aa740efb004579574a0e1e2 2023-04-03 05:54:21,703 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=466, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/e08857f97e674e8d815ed38654c73b0f 2023-04-03 05:54:22,104 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/91a36583f0164fabbb7648bb022209d1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/91a36583f0164fabbb7648bb022209d1 2023-04-03 05:54:22,109 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/C of 574b09e98d501def4bb65b12ef4de07a into 91a36583f0164fabbb7648bb022209d1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:22,109 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:22,109 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/C, priority=12, startTime=1680501261607; duration=0sec 2023-04-03 05:54:22,109 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:22,121 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/485e7ecf40d141209f5f3b7bacb95676 2023-04-03 05:54:22,487 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/88269a9e925c413fa1765da8e9b3cee9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/88269a9e925c413fa1765da8e9b3cee9 2023-04-03 05:54:22,492 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/A of 574b09e98d501def4bb65b12ef4de07a into 88269a9e925c413fa1765da8e9b3cee9(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:22,492 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:22,492 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/A, priority=10, startTime=1680501261607; duration=0sec 2023-04-03 05:54:22,492 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:22,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:22,500 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:22,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:22,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501322507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:22,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501322510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:22,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501322527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,532 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=466 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/f3f013fe4553457fa910a1c52e32a165 2023-04-03 05:54:22,538 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/e08857f97e674e8d815ed38654c73b0f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/e08857f97e674e8d815ed38654c73b0f 2023-04-03 05:54:22,542 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/e08857f97e674e8d815ed38654c73b0f, entries=150, sequenceid=466, filesize=25.4 K 2023-04-03 05:54:22,545 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/485e7ecf40d141209f5f3b7bacb95676 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/485e7ecf40d141209f5f3b7bacb95676 2023-04-03 05:54:22,554 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/485e7ecf40d141209f5f3b7bacb95676, entries=150, sequenceid=466, filesize=11.9 K 2023-04-03 05:54:22,555 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/f3f013fe4553457fa910a1c52e32a165 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f3f013fe4553457fa910a1c52e32a165 2023-04-03 05:54:22,562 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f3f013fe4553457fa910a1c52e32a165, entries=150, sequenceid=466, filesize=11.9 K 2023-04-03 05:54:22,563 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 574b09e98d501def4bb65b12ef4de07a in 932ms, sequenceid=466, compaction requested=false 2023-04-03 05:54:22,563 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:22,563 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:22,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:22,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:22,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:22,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:22,565 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,565 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:22,565 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,565 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:22,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:22,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:22,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dd1fcb2[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:22,568 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:22,568 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,570 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:22,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:22,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:22,571 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,571 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:22,573 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:22,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:22,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:22,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:22,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:22,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:22,577 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:22,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:22,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:22,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:22,577 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:22,579 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,579 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,579 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:22,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:22,580 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:22,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:22,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:22,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:22,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:22,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:22,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:22,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:22,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:22,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:22,589 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:22,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:22,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:22,589 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:22,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:22,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:22,613 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 05:54:22,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:22,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:22,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:22,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:22,613 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:22,613 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:22,630 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040383db8b9919c04261b470869c3865913e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040383db8b9919c04261b470869c3865913e 2023-04-03 05:54:22,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:22,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501322637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:22,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501322638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,640 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:22,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501322639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,641 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=481, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/7e8f7c4494bb41849c528fc5350d70c5 2023-04-03 05:54:22,689 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:22,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:22,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:22,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:22,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:22,693 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:22,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:22,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:22,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:22,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,695 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,695 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:22,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:22,696 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,696 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:22,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:22,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:22,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:22,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:22,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:22,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:22,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:22,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:22,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:22,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:22,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:22,698 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:22,698 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:22,698 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,698 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:22,698 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:22,699 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,699 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:22,699 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,699 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a391f35[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:22,701 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:22,701 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,702 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:22,702 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:22,703 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,703 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:22,705 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:22,705 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:22,705 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:22,705 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:22,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:22,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:22,708 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:22,708 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:22,708 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:22,708 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:22,708 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:22,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:22,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:22,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:22,711 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,712 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:22,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:22,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:22,716 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:22,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:22,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:22,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:22,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:22,716 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:22,716 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:22,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:22,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53244 deadline: 1680501322739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:22,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501322739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:22,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501322741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,816 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:22,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:22,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:22,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:22,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:22,820 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:22,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:22,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:22,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:22,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,822 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,822 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:22,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:22,822 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,822 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:22,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,823 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:22,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:22,823 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:22,823 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:22,823 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:22,823 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:22,823 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:22,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:22,824 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:22,824 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:22,824 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:22,824 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:22,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:22,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:22,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:22,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:22,827 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:22,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:22,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5167e727[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:22,828 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:22,828 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,829 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,829 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,830 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:22,830 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:22,830 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:22,830 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:22,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,831 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:22,831 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:22,831 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:22,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:22,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:22,833 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:22,833 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:22,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:22,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:22,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:22,833 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:22,835 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,835 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,835 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:22,835 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:22,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:22,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:22,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,841 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:22,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:22,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:22,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:22,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:22,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:22,845 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:22,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:22,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,846 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:22,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,846 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:22,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:22,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:22,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:22,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501322941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,943 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:22,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501322941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:22,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501322942, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,946 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:22,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:22,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:22,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:22,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:22,949 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:22,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:22,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:22,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:22,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,950 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,950 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:22,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:22,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:22,951 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,951 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:22,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,951 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:22,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:22,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:22,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:22,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:22,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:22,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:22,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:22,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:22,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:22,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:22,955 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:22,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:22,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:22,956 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:22,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a58c161[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:22,956 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:22,958 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:22,958 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,958 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:22,960 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,960 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:22,960 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:22,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:22,960 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:22,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:22,963 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:22,963 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:22,963 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:22,963 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:22,963 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:22,964 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,964 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,964 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:22,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:22,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,965 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:22,966 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:22,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:22,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:22,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:22,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:22,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:22,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:22,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:22,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:22,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,969 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:22,969 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:22,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:22,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:22,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:22,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:22,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:22,969 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:23,053 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/12896c09ca1f4a9ba3f711f9bf72bd3e 2023-04-03 05:54:23,069 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:23,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:23,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:23,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:23,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:23,073 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:23,073 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:23,073 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:23,073 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:23,073 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,073 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,074 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,074 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:23,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:23,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:23,074 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,075 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:23,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:23,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:23,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:23,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:23,076 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:23,076 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:23,076 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:23,076 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:23,076 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:23,083 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,083 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:23,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:23,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:23,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:23,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@84e9cf4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:23,085 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:23,085 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,087 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:23,087 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:23,087 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,087 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:23,088 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,088 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:23,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:23,088 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:23,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:23,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:23,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:23,091 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:23,091 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:23,091 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:23,091 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:23,091 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:23,093 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,093 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,093 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:23,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:23,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:23,093 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:23,094 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:23,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:23,098 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:23,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:23,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:23,098 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:23,098 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:23,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:23,098 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:23,098 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:23,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:23,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:23,199 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:23,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:23,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:23,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:23,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:23,202 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:23,202 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:23,202 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:23,202 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:23,202 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,202 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,203 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:23,203 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,204 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,205 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,205 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:23,205 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:23,205 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:23,205 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:23,205 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:23,205 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:23,205 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:23,205 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:23,205 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:23,205 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:23,206 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:23,206 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:23,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:23,207 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,207 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:23,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:23,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:23,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a7cea7f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:23,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:23,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,211 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,211 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:23,211 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:23,211 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:23,211 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,211 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:23,213 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,213 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:23,214 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:23,214 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:23,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:23,216 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:23,216 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:23,216 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:23,216 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:23,216 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:23,216 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:23,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:23,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:23,218 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:23,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,220 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:23,222 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:23,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:23,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:23,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:23,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:23,222 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:23,222 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:23,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:23,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:23,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501323244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:23,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501323245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,248 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:23,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501323245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,323 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:23,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:23,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:23,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:23,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:23,327 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:23,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:23,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:23,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:23,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,330 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:23,330 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,330 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,330 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:23,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,331 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:23,336 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:23,336 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:23,336 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:23,336 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:23,336 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:23,336 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:23,336 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:23,336 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:23,336 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:23,336 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:23,337 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:23,337 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:23,337 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:23,337 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,337 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:23,337 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:23,338 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:23,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,339 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:23,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:23,340 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:23,340 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14cc74c8[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:23,340 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,341 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,342 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,342 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:23,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:23,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:23,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:23,343 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:23,343 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:23,343 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,343 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:23,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:23,347 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:23,347 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:23,347 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:23,347 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:23,347 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:23,349 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,350 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:23,350 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:23,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:23,350 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,351 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:23,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:23,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:23,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:23,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:23,354 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:23,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,354 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:23,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:23,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:23,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:23,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:23,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:23,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:23,454 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:23,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:23,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:23,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:23,458 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:23,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:23,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:23,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:23,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,459 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:23,459 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:23,460 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,460 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:23,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:23,461 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:23,461 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:23,461 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:23,463 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/cdaf28ce082d4dd082d95ab903010bec 2023-04-03 05:54:23,463 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:23,463 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:23,463 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:23,463 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:23,465 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,465 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/7e8f7c4494bb41849c528fc5350d70c5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/7e8f7c4494bb41849c528fc5350d70c5 2023-04-03 05:54:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:23,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:23,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7873ecf9[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:23,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:23,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/7e8f7c4494bb41849c528fc5350d70c5, entries=200, sequenceid=481, filesize=32.3 K 2023-04-03 05:54:23,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/12896c09ca1f4a9ba3f711f9bf72bd3e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/12896c09ca1f4a9ba3f711f9bf72bd3e 2023-04-03 05:54:23,473 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:23,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:23,473 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,473 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:23,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:23,475 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:23,475 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:23,475 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:23,475 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/12896c09ca1f4a9ba3f711f9bf72bd3e, entries=150, sequenceid=481, filesize=11.9 K 2023-04-03 05:54:23,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:23,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:23,478 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:23,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:23,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:23,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:23,478 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:23,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:23,480 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,480 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/cdaf28ce082d4dd082d95ab903010bec as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cdaf28ce082d4dd082d95ab903010bec 2023-04-03 05:54:23,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:23,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:23,484 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:23,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:23,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:23,484 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:23,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:23,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:23,484 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:23,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:23,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cdaf28ce082d4dd082d95ab903010bec, entries=150, sequenceid=481, filesize=11.9 K 2023-04-03 05:54:23,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:23,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:23,485 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 574b09e98d501def4bb65b12ef4de07a in 872ms, sequenceid=481, compaction requested=true 2023-04-03 05:54:23,486 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:23,486 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:23,486 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:23,486 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:23,486 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:23,486 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:23,487 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85826 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:23,487 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/A is initiating minor compaction (all files) 2023-04-03 05:54:23,487 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:23,487 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/A in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:23,487 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/B is initiating minor compaction (all files) 2023-04-03 05:54:23,487 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/88269a9e925c413fa1765da8e9b3cee9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/e08857f97e674e8d815ed38654c73b0f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/7e8f7c4494bb41849c528fc5350d70c5] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=83.8 K 2023-04-03 05:54:23,487 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/B in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:23,487 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/ca028b3d0eec450cb2d992fdba2ec433, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/485e7ecf40d141209f5f3b7bacb95676, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/12896c09ca1f4a9ba3f711f9bf72bd3e] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=36.4 K 2023-04-03 05:54:23,488 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 88269a9e925c413fa1765da8e9b3cee9, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680501260251 2023-04-03 05:54:23,488 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ca028b3d0eec450cb2d992fdba2ec433, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680501260251 2023-04-03 05:54:23,488 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e08857f97e674e8d815ed38654c73b0f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1680501260368 2023-04-03 05:54:23,488 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 485e7ecf40d141209f5f3b7bacb95676, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1680501260368 2023-04-03 05:54:23,488 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 7e8f7c4494bb41849c528fc5350d70c5, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1680501262503 2023-04-03 05:54:23,488 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 12896c09ca1f4a9ba3f711f9bf72bd3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1680501262503 2023-04-03 05:54:23,496 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#B#compaction#189 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:23,505 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#A#compaction#188 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:23,525 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/19d03acb357b436bbf02be3ddc522c35 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/19d03acb357b436bbf02be3ddc522c35 2023-04-03 05:54:23,531 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/B of 574b09e98d501def4bb65b12ef4de07a into 19d03acb357b436bbf02be3ddc522c35(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:23,532 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:23,532 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/B, priority=13, startTime=1680501263486; duration=0sec 2023-04-03 05:54:23,532 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:23,532 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:23,533 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:23,533 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/C is initiating minor compaction (all files) 2023-04-03 05:54:23,534 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/C in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:23,534 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/91a36583f0164fabbb7648bb022209d1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f3f013fe4553457fa910a1c52e32a165, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cdaf28ce082d4dd082d95ab903010bec] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=36.4 K 2023-04-03 05:54:23,534 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 91a36583f0164fabbb7648bb022209d1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680501260251 2023-04-03 05:54:23,535 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting f3f013fe4553457fa910a1c52e32a165, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=466, earliestPutTs=1680501260368 2023-04-03 05:54:23,535 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting cdaf28ce082d4dd082d95ab903010bec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1680501262503 2023-04-03 05:54:23,547 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#C#compaction#190 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:23,581 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/811315447c3e43df8b2bc766a4a547bc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/811315447c3e43df8b2bc766a4a547bc 2023-04-03 05:54:23,585 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:23,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:23,585 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/C of 574b09e98d501def4bb65b12ef4de07a into 811315447c3e43df8b2bc766a4a547bc(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:23,586 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:23,586 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/C, priority=13, startTime=1680501263486; duration=0sec 2023-04-03 05:54:23,586 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:23,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:23,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:23,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:23,592 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:23,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:23,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:23,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:23,593 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,593 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:23,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:23,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:23,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:23,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:23,598 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/23773312f3594ab3883910637c1397fe as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/23773312f3594ab3883910637c1397fe 2023-04-03 05:54:23,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:23,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:23,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:23,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:23,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:23,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:23,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:23,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:23,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:23,606 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 05:54:23,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:23,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:23,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:23,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:23,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:23,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:23,608 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/A of 574b09e98d501def4bb65b12ef4de07a into 23773312f3594ab3883910637c1397fe(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:23,608 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:23,608 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/A, priority=13, startTime=1680501263486; duration=0sec 2023-04-03 05:54:23,608 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:23,622 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304031104d04919c34e9fb22dda4c3732b5b4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031104d04919c34e9fb22dda4c3732b5b4 2023-04-03 05:54:23,631 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=503, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/5abcdcfcb7de4fc2a180c49a24683176 2023-04-03 05:54:23,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:23,760 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:23,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:23,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501323768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:23,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501323768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:23,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501323769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:23,874 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:23,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501323874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501323874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:23,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:23,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501323874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,041 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/0f9f8f874a7547de864db1575b91c5d1 2023-04-03 05:54:24,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:24,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501324075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:24,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501324076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,077 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:24,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501324076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,378 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:24,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501324378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,379 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:24,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501324379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:24,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501324380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,452 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=503 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/6af2296bc51343b39fea3048ab83c82a 2023-04-03 05:54:24,457 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/5abcdcfcb7de4fc2a180c49a24683176 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/5abcdcfcb7de4fc2a180c49a24683176 2023-04-03 05:54:24,460 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/5abcdcfcb7de4fc2a180c49a24683176, entries=150, sequenceid=503, filesize=25.4 K 2023-04-03 05:54:24,461 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/0f9f8f874a7547de864db1575b91c5d1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0f9f8f874a7547de864db1575b91c5d1 2023-04-03 05:54:24,465 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0f9f8f874a7547de864db1575b91c5d1, entries=150, sequenceid=503, filesize=11.9 K 2023-04-03 05:54:24,466 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/6af2296bc51343b39fea3048ab83c82a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6af2296bc51343b39fea3048ab83c82a 2023-04-03 05:54:24,469 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6af2296bc51343b39fea3048ab83c82a, entries=150, sequenceid=503, filesize=11.9 K 2023-04-03 05:54:24,470 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 574b09e98d501def4bb65b12ef4de07a in 864ms, sequenceid=503, compaction requested=false 2023-04-03 05:54:24,470 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:24,470 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:24,471 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:24,471 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:24,471 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,471 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:24,471 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:24,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:24,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:24,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:24,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:24,474 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6663f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:24,474 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:24,474 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:24,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:24,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:24,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:24,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:24,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:24,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:24,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:24,482 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:24,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:24,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:24,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:24,482 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:24,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:24,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:24,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,484 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,485 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:24,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:24,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:24,487 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:24,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:24,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:24,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:24,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:24,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,488 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,488 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:24,488 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:24,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:24,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:24,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:24,541 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:54:24,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:24,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:24,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:24,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:24,541 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:24,541 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:24,588 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:24,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:24,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:24,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:24,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:24,591 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:24,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:24,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:24,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:24,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:24,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:24,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:24,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:24,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:24,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:24,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501324595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,602 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:24,602 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:24,602 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:24,602 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:24,602 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:24,602 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:24,602 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:24,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,604 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:24,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:24,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:24,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19944bc6[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:24,606 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:24,606 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,607 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:24,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:24,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:24,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:24,609 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:24,609 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:24,612 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:24,612 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:24,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:24,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:24,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:24,612 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:24,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:24,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:24,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,616 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,617 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,619 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:24,619 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,619 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:24,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:24,619 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:24,620 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:24,619 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:24,620 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:24,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:24,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:24,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:24,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:24,699 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:24,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501324699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,720 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:24,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:24,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:24,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:24,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:24,732 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:24,732 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:24,732 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:24,732 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:24,732 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,733 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,734 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:24,734 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:24,734 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,734 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,735 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:24,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:24,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:24,736 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:24,736 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:24,736 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:24,736 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:24,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:24,745 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,745 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:24,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:24,746 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,746 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:24,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:24,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:24,747 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e890608[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:24,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:24,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,749 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,749 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:24,749 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:24,749 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,749 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:24,749 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:24,754 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:24,754 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,754 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:24,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:24,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:24,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:24,757 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:24,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:24,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:24,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:24,757 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:24,759 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,759 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:24,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,759 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:24,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:24,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:24,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,761 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,763 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:24,763 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:24,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:24,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:24,763 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:24,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:24,763 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:24,764 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:24,764 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:24,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:24,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:24,864 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:24,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:24,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:24,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:24,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:24,868 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:24,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:24,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:24,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:24,869 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,869 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:24,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:24,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:24,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:24,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:24,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:24,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501324881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:24,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:24,884 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:24,884 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:24,884 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:24,884 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:24,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:24,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:24,885 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,885 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:24,885 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:24,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:24,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501324882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:24,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501324884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,886 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:24,887 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:24,887 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,887 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:24,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:24,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9fa2b04[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:24,889 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:24,889 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,890 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,890 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:24,890 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:24,890 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:24,891 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,891 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:24,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:24,892 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:24,892 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:24,892 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:24,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:24,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:24,895 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:24,895 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:24,896 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:24,896 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:24,896 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:24,896 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:24,900 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,900 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:24,901 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:24,901 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:24,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:24,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501324901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,903 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:24,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:24,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:24,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:24,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:24,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:24,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:24,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:24,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:24,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:24,905 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:24,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:24,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:24,905 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:24,906 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:24,906 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:24,963 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304036f506ec39ef44f50a1e83002b81055f5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036f506ec39ef44f50a1e83002b81055f5 2023-04-03 05:54:25,006 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:25,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:25,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:25,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:25,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:25,009 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:25,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:25,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:25,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:25,010 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,010 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,011 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,011 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:25,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:25,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:25,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:25,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:25,013 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:25,013 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:25,013 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:25,013 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:25,013 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:25,013 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:25,013 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:25,013 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:25,013 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:25,013 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:25,014 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:25,014 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:25,014 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:25,014 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,014 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:25,014 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:25,016 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:25,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:25,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:25,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:25,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eda339a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:25,018 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:25,018 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:25,021 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:25,021 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:25,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:25,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:25,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:25,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:25,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:25,024 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:25,024 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:25,024 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:25,024 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:25,024 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:25,026 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,026 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:25,026 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:25,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:25,027 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,028 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,039 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,039 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:25,039 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,039 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:25,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:25,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:25,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:25,039 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:25,039 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,040 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,040 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:25,040 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:25,040 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,040 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,040 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:25,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:25,140 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:25,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:25,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:25,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:25,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:25,143 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:25,144 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:25,144 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:25,144 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:25,144 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,144 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,147 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:25,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:25,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:25,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:25,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:25,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:25,149 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:25,149 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:25,149 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:25,149 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:25,149 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:25,149 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:25,149 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:25,149 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:25,150 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:25,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:25,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:25,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:25,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:25,153 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:25,153 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,153 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b85fc[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:25,154 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:25,155 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:25,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:25,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:25,156 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,156 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:25,157 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,157 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:25,157 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:25,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,157 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:25,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:25,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:25,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:25,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:25,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:25,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:25,159 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:25,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:25,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:25,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:25,159 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:25,162 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,162 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:25,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,162 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:25,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:25,162 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,164 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:25,168 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:25,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:25,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:25,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,169 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,169 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:25,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:25,169 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:25,169 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:25,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:25,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:25,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:25,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:25,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501325204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,269 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:25,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:25,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:25,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:25,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:25,273 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:25,273 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:25,273 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:25,273 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:25,273 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,273 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,275 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:25,275 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,275 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,275 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:25,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:25,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:25,277 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:25,277 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:25,277 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:25,277 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:25,277 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:25,277 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:25,277 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:25,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:25,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:25,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,280 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:25,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:25,280 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,280 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:25,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23347a17[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:25,282 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:25,282 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,283 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:25,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:25,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:25,284 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,284 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:25,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:25,286 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:25,286 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:25,286 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:25,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:25,289 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:25,289 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:25,289 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:25,289 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:25,289 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:25,291 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,291 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:25,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,291 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:25,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:25,292 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:25,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,293 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,296 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:25,296 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:25,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:25,296 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,296 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,297 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:25,297 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:25,297 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,297 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:25,297 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,297 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:25,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:25,297 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,297 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:25,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:25,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:25,369 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=518, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/0847547187e84db39c7d34ab0afd7b56 2023-04-03 05:54:25,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/cbb8599d4c9d43bf9751343b6f809b70 2023-04-03 05:54:25,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=518 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/2191cb783f9148db9790e8053d680a28 2023-04-03 05:54:25,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/0847547187e84db39c7d34ab0afd7b56 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/0847547187e84db39c7d34ab0afd7b56 2023-04-03 05:54:25,397 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:25,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:25,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/0847547187e84db39c7d34ab0afd7b56, entries=150, sequenceid=518, filesize=25.4 K 2023-04-03 05:54:25,399 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/cbb8599d4c9d43bf9751343b6f809b70 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/cbb8599d4c9d43bf9751343b6f809b70 2023-04-03 05:54:25,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:25,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:25,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:25,400 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:25,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:25,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:25,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:25,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,402 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:25,402 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:25,403 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:25,403 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:25,404 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:25,404 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:25,404 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:25,404 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,404 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:25,406 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/cbb8599d4c9d43bf9751343b6f809b70, entries=150, sequenceid=518, filesize=11.9 K 2023-04-03 05:54:25,407 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/2191cb783f9148db9790e8053d680a28 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/2191cb783f9148db9790e8053d680a28 2023-04-03 05:54:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:25,408 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:25,408 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:25,408 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:25,408 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:25,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:25,410 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,410 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:25,410 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,410 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:25,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:25,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:25,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bcec77e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:25,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:25,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,414 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,414 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:25,414 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:25,414 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:25,415 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/2191cb783f9148db9790e8053d680a28, entries=150, sequenceid=518, filesize=11.9 K 2023-04-03 05:54:25,415 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,415 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:25,415 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 574b09e98d501def4bb65b12ef4de07a in 874ms, sequenceid=518, compaction requested=true 2023-04-03 05:54:25,416 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:25,416 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:25,416 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:25,416 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:25,416 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:25,416 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:25,416 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:25,416 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:25,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:25,417 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78888 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:25,417 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/A is initiating minor compaction (all files) 2023-04-03 05:54:25,417 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/A in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:25,417 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/23773312f3594ab3883910637c1397fe, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/5abcdcfcb7de4fc2a180c49a24683176, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/0847547187e84db39c7d34ab0afd7b56] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=77.0 K 2023-04-03 05:54:25,418 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:25,418 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/B is initiating minor compaction (all files) 2023-04-03 05:54:25,418 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/B in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:25,418 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/19d03acb357b436bbf02be3ddc522c35, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0f9f8f874a7547de864db1575b91c5d1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/cbb8599d4c9d43bf9751343b6f809b70] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=36.5 K 2023-04-03 05:54:25,418 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 23773312f3594ab3883910637c1397fe, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1680501262503 2023-04-03 05:54:25,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:25,418 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 5abcdcfcb7de4fc2a180c49a24683176, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1680501262625 2023-04-03 05:54:25,418 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 19d03acb357b436bbf02be3ddc522c35, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1680501262503 2023-04-03 05:54:25,419 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 0847547187e84db39c7d34ab0afd7b56, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680501263766 2023-04-03 05:54:25,420 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0f9f8f874a7547de864db1575b91c5d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1680501262625 2023-04-03 05:54:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,420 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting cbb8599d4c9d43bf9751343b6f809b70, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680501263766 2023-04-03 05:54:25,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:25,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:25,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:25,426 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:25,426 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:25,426 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:25,426 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:25,426 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:25,429 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,429 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:25,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:25,430 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:25,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:25,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,432 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:25,435 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:25,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,437 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:25,437 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:25,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:25,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:25,438 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,438 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,438 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,438 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,438 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,438 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:25,438 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:25,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:25,438 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:25,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:25,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:25,439 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:25,439 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:25,441 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#B#compaction#197 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:25,459 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#A#compaction#198 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:25,519 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/d26dfdbd41734c418cc7233de3fec63d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/d26dfdbd41734c418cc7233de3fec63d 2023-04-03 05:54:25,526 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/B of 574b09e98d501def4bb65b12ef4de07a into d26dfdbd41734c418cc7233de3fec63d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:25,526 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:25,526 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/B, priority=13, startTime=1680501265416; duration=0sec 2023-04-03 05:54:25,526 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:25,526 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:25,529 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:25,529 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/C is initiating minor compaction (all files) 2023-04-03 05:54:25,529 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/C in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:25,529 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/811315447c3e43df8b2bc766a4a547bc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6af2296bc51343b39fea3048ab83c82a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/2191cb783f9148db9790e8053d680a28] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=36.5 K 2023-04-03 05:54:25,529 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 811315447c3e43df8b2bc766a4a547bc, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1680501262503 2023-04-03 05:54:25,530 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6af2296bc51343b39fea3048ab83c82a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=503, earliestPutTs=1680501262625 2023-04-03 05:54:25,530 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 2191cb783f9148db9790e8053d680a28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680501263766 2023-04-03 05:54:25,539 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:25,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:25,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:25,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:25,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:25,556 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:25,557 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:25,557 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:25,557 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:25,557 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,557 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,558 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:25,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:25,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:25,558 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,559 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,559 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:25,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:25,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:25,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:25,569 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#C#compaction#199 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:25,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:25,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:25,578 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:25,578 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:25,578 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:25,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:25,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:25,579 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:25,580 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:25,580 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 05:54:25,580 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:25,580 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:25,580 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:25,580 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:25,580 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:25,580 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:25,588 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/395e3e4d54fe4d42a8329ebcc03e7d65 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/395e3e4d54fe4d42a8329ebcc03e7d65 2023-04-03 05:54:25,590 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/548c915bf0aa4f51856654a5a177d34a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/548c915bf0aa4f51856654a5a177d34a 2023-04-03 05:54:25,594 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/A of 574b09e98d501def4bb65b12ef4de07a into 395e3e4d54fe4d42a8329ebcc03e7d65(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:25,594 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:25,594 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/A, priority=13, startTime=1680501265416; duration=0sec 2023-04-03 05:54:25,594 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:25,598 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/C of 574b09e98d501def4bb65b12ef4de07a into 548c915bf0aa4f51856654a5a177d34a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:25,598 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:25,598 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/C, priority=13, startTime=1680501265416; duration=0sec 2023-04-03 05:54:25,599 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:25,606 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040349ebb56aa15541fd82ca72591c723ea5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040349ebb56aa15541fd82ca72591c723ea5 2023-04-03 05:54:25,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:25,709 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:25,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:25,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501325728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,832 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:25,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501325830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:25,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501325887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:25,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501325887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:25,900 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:25,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501325899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,018 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=537, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/0afb273c51b94fefac5df55a10fad33c 2023-04-03 05:54:26,030 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/01b2402fc8ae44dfa8be36272d13b052 2023-04-03 05:54:26,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:26,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501326041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,053 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=537 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/c917b830219e486eb939dc9ebe141385 2023-04-03 05:54:26,058 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/0afb273c51b94fefac5df55a10fad33c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/0afb273c51b94fefac5df55a10fad33c 2023-04-03 05:54:26,067 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/0afb273c51b94fefac5df55a10fad33c, entries=150, sequenceid=537, filesize=25.4 K 2023-04-03 05:54:26,068 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/01b2402fc8ae44dfa8be36272d13b052 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/01b2402fc8ae44dfa8be36272d13b052 2023-04-03 05:54:26,073 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/01b2402fc8ae44dfa8be36272d13b052, entries=150, sequenceid=537, filesize=11.9 K 2023-04-03 05:54:26,073 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/c917b830219e486eb939dc9ebe141385 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/c917b830219e486eb939dc9ebe141385 2023-04-03 05:54:26,078 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/c917b830219e486eb939dc9ebe141385, entries=150, sequenceid=537, filesize=11.9 K 2023-04-03 05:54:26,079 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 574b09e98d501def4bb65b12ef4de07a in 499ms, sequenceid=537, compaction requested=false 2023-04-03 05:54:26,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:26,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:26,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:26,081 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:26,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:26,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:26,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:26,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:26,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14677d7e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:26,083 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:26,083 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:26,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:26,085 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,085 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:26,087 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:26,087 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:26,087 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,087 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:26,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:26,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:26,089 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:26,090 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:26,090 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:26,090 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:26,090 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:26,091 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,091 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:26,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,091 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:26,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:26,092 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:26,093 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:26,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,099 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,099 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:26,099 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:26,099 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:26,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:26,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:26,099 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:26,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:26,100 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:26,099 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:26,100 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,100 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,100 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,100 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,100 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:26,100 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:26,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:26,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:26,200 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:26,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:26,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:26,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:26,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:26,203 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:26,203 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:26,203 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:26,203 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:26,204 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,204 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:26,206 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:26,206 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:26,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:26,206 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,206 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:26,207 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:26,207 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:26,207 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:26,207 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:26,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:26,208 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:26,208 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:26,208 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:54:26,208 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:26,208 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:26,208 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:26,208 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:26,208 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:26,208 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:26,224 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040309a87584302b4676b570be11be2c1a71 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040309a87584302b4676b570be11be2c1a71 2023-04-03 05:54:26,231 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=553, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/225c06f31a12416d88c3aa64cd7b5415 2023-04-03 05:54:26,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:26,350 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:26,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:26,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501326383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:26,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501326486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,650 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/3518ae4b7d994ee8bd2c8b02c37eaaea 2023-04-03 05:54:26,664 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=553 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/3367c1294bb6434aba67b54f2ac8b5ea 2023-04-03 05:54:26,668 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/225c06f31a12416d88c3aa64cd7b5415 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/225c06f31a12416d88c3aa64cd7b5415 2023-04-03 05:54:26,672 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/225c06f31a12416d88c3aa64cd7b5415, entries=150, sequenceid=553, filesize=25.4 K 2023-04-03 05:54:26,673 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/3518ae4b7d994ee8bd2c8b02c37eaaea as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/3518ae4b7d994ee8bd2c8b02c37eaaea 2023-04-03 05:54:26,678 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/3518ae4b7d994ee8bd2c8b02c37eaaea, entries=150, sequenceid=553, filesize=11.9 K 2023-04-03 05:54:26,679 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/3367c1294bb6434aba67b54f2ac8b5ea as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/3367c1294bb6434aba67b54f2ac8b5ea 2023-04-03 05:54:26,683 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/3367c1294bb6434aba67b54f2ac8b5ea, entries=150, sequenceid=553, filesize=11.9 K 2023-04-03 05:54:26,685 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 574b09e98d501def4bb65b12ef4de07a in 477ms, sequenceid=553, compaction requested=true 2023-04-03 05:54:26,685 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:26,685 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:26,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:26,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:26,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:26,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:26,687 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,687 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:26,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:26,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:26,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d3a1e6b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:26,689 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:26,689 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:26,691 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 05:54:26,691 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:26,691 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,691 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:26,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:26,692 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:26,691 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:26,692 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:26,692 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:26,692 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:26,693 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:26,693 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:26,693 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:26,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:26,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:26,694 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:26,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:26,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:26,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:26,694 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:26,696 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:26,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:26,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:26,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,706 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:26,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:26,707 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403a39f176d8e7b4d20a5bc32be0f3ae600 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a39f176d8e7b4d20a5bc32be0f3ae600 2023-04-03 05:54:26,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:26,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:26,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:26,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:26,711 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:26,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:26,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:26,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:26,711 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:26,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:26,712 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,712 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:26,712 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:26,716 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:26,716 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:26,718 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=572, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/cf8e2fa2c9b64bf7ab39552fe397041d 2023-04-03 05:54:26,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:26,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501326733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,751 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/bbb20a22a0b24f1dbf1b7a59c23b094f 2023-04-03 05:54:26,778 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/f64fb35fb9c246e8ac660e1416ab36f7 2023-04-03 05:54:26,783 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/cf8e2fa2c9b64bf7ab39552fe397041d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/cf8e2fa2c9b64bf7ab39552fe397041d 2023-04-03 05:54:26,787 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/cf8e2fa2c9b64bf7ab39552fe397041d, entries=200, sequenceid=572, filesize=32.3 K 2023-04-03 05:54:26,788 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/bbb20a22a0b24f1dbf1b7a59c23b094f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/bbb20a22a0b24f1dbf1b7a59c23b094f 2023-04-03 05:54:26,792 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/bbb20a22a0b24f1dbf1b7a59c23b094f, entries=150, sequenceid=572, filesize=11.9 K 2023-04-03 05:54:26,793 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/f64fb35fb9c246e8ac660e1416ab36f7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f64fb35fb9c246e8ac660e1416ab36f7 2023-04-03 05:54:26,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f64fb35fb9c246e8ac660e1416ab36f7, entries=150, sequenceid=572, filesize=11.9 K 2023-04-03 05:54:26,802 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 574b09e98d501def4bb65b12ef4de07a in 111ms, sequenceid=572, compaction requested=true 2023-04-03 05:54:26,802 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:26,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:26,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:26,802 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:26,802 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:26,802 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:26,803 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 112076 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:26,803 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:26,803 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/A is initiating minor compaction (all files) 2023-04-03 05:54:26,804 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/B is initiating minor compaction (all files) 2023-04-03 05:54:26,804 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/A in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:26,804 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/B in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:26,804 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/395e3e4d54fe4d42a8329ebcc03e7d65, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/0afb273c51b94fefac5df55a10fad33c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/225c06f31a12416d88c3aa64cd7b5415, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/cf8e2fa2c9b64bf7ab39552fe397041d] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=109.4 K 2023-04-03 05:54:26,804 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/d26dfdbd41734c418cc7233de3fec63d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/01b2402fc8ae44dfa8be36272d13b052, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/3518ae4b7d994ee8bd2c8b02c37eaaea, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/bbb20a22a0b24f1dbf1b7a59c23b094f] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=48.5 K 2023-04-03 05:54:26,804 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 395e3e4d54fe4d42a8329ebcc03e7d65, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680501263766 2023-04-03 05:54:26,804 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting d26dfdbd41734c418cc7233de3fec63d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680501263766 2023-04-03 05:54:26,804 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 0afb273c51b94fefac5df55a10fad33c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680501264583 2023-04-03 05:54:26,805 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 01b2402fc8ae44dfa8be36272d13b052, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680501264583 2023-04-03 05:54:26,805 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 225c06f31a12416d88c3aa64cd7b5415, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680501265719 2023-04-03 05:54:26,805 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3518ae4b7d994ee8bd2c8b02c37eaaea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680501265719 2023-04-03 05:54:26,805 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting cf8e2fa2c9b64bf7ab39552fe397041d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680501266378 2023-04-03 05:54:26,806 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting bbb20a22a0b24f1dbf1b7a59c23b094f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680501266378 2023-04-03 05:54:26,816 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:26,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:26,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:26,819 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#B#compaction#210 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:26,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:26,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:26,824 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:26,824 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:26,824 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:26,824 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:26,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:26,827 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,827 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:26,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:26,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:26,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:26,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:26,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:26,828 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,828 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:26,835 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#A#compaction#209 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:26,837 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:26,837 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:26,838 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:26,838 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:26,838 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:26,838 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:26,838 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:26,839 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:26,839 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:26,839 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 05:54:26,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:26,839 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:26,839 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:26,840 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:26,840 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:26,840 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:26,840 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:26,840 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:26,865 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/de12715bb6284f05939a9d9e8c24d5f2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/de12715bb6284f05939a9d9e8c24d5f2 2023-04-03 05:54:26,897 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/B of 574b09e98d501def4bb65b12ef4de07a into de12715bb6284f05939a9d9e8c24d5f2(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:26,897 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:26,897 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/B, priority=12, startTime=1680501266802; duration=0sec 2023-04-03 05:54:26,897 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:26,897 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:26,899 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:26,899 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/C is initiating minor compaction (all files) 2023-04-03 05:54:26,899 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/C in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:26,899 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/548c915bf0aa4f51856654a5a177d34a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/c917b830219e486eb939dc9ebe141385, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/3367c1294bb6434aba67b54f2ac8b5ea, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f64fb35fb9c246e8ac660e1416ab36f7] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=48.5 K 2023-04-03 05:54:26,899 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 548c915bf0aa4f51856654a5a177d34a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=518, earliestPutTs=1680501263766 2023-04-03 05:54:26,900 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting c917b830219e486eb939dc9ebe141385, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=537, earliestPutTs=1680501264583 2023-04-03 05:54:26,900 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3367c1294bb6434aba67b54f2ac8b5ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=553, earliestPutTs=1680501265719 2023-04-03 05:54:26,901 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting f64fb35fb9c246e8ac660e1416ab36f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680501266378 2023-04-03 05:54:26,933 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#C#compaction#212 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:26,954 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/59b7952a08ef46c698a4e34efb89dad2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/59b7952a08ef46c698a4e34efb89dad2 2023-04-03 05:54:26,956 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:26,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501326952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:26,957 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/26ca58f6915d47eeac660ffa1d6a9969 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/26ca58f6915d47eeac660ffa1d6a9969 2023-04-03 05:54:26,962 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/A of 574b09e98d501def4bb65b12ef4de07a into 59b7952a08ef46c698a4e34efb89dad2(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:26,962 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:26,962 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/A, priority=12, startTime=1680501266802; duration=0sec 2023-04-03 05:54:26,963 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:26,964 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/C of 574b09e98d501def4bb65b12ef4de07a into 26ca58f6915d47eeac660ffa1d6a9969(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:26,964 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:26,964 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/C, priority=12, startTime=1680501266802; duration=0sec 2023-04-03 05:54:26,964 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:27,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:27,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501327057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,265 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:27,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501327261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,305 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403548a8e53fea841a2aeb52ea40fa81a62 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403548a8e53fea841a2aeb52ea40fa81a62 2023-04-03 05:54:27,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:27,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501327566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,714 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=586, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/1c4da99d702a45bfa14e14cd2e754071 2023-04-03 05:54:27,728 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/44b6dac4d24a4d77822bda1e28cf7289 2023-04-03 05:54:27,754 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/6db3c8eecc8b42368414e2c3611c0927 2023-04-03 05:54:27,758 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/1c4da99d702a45bfa14e14cd2e754071 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/1c4da99d702a45bfa14e14cd2e754071 2023-04-03 05:54:27,762 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/1c4da99d702a45bfa14e14cd2e754071, entries=200, sequenceid=586, filesize=32.3 K 2023-04-03 05:54:27,763 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/44b6dac4d24a4d77822bda1e28cf7289 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/44b6dac4d24a4d77822bda1e28cf7289 2023-04-03 05:54:27,767 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/44b6dac4d24a4d77822bda1e28cf7289, entries=150, sequenceid=586, filesize=11.9 K 2023-04-03 05:54:27,768 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/6db3c8eecc8b42368414e2c3611c0927 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6db3c8eecc8b42368414e2c3611c0927 2023-04-03 05:54:27,772 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6db3c8eecc8b42368414e2c3611c0927, entries=150, sequenceid=586, filesize=11.9 K 2023-04-03 05:54:27,772 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 574b09e98d501def4bb65b12ef4de07a in 933ms, sequenceid=586, compaction requested=false 2023-04-03 05:54:27,773 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:27,773 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:27,773 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:27,773 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:27,773 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,773 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:27,773 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:27,775 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:27,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:27,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:27,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dde3119[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:27,779 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:27,779 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,780 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,780 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:27,780 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:27,780 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:27,780 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,780 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:27,781 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:27,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:27,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:27,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:27,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:27,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:27,784 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:27,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:27,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:27,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:27,784 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:27,786 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,786 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,787 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:27,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:27,787 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:27,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,792 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,792 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:27,792 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:27,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:27,792 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:27,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:27,792 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:27,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:27,792 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:27,792 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:27,792 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:27,792 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:27,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:27,792 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,792 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,792 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,792 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:27,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:27,892 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:27,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:27,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:27,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:27,901 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 05:54:27,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:27,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:27,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:27,902 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:27,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:27,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:27,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:27,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:27,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:27,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:27,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:27,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:27,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:27,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:27,905 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:27,905 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,905 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:27,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:27,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:27,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:27,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:27,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:27,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:27,909 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:27,910 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:27,914 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:27,914 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:27,914 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:27,914 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:27,914 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:27,914 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:27,914 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:27,914 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:27,914 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,916 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:27,916 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:27,918 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,918 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:27,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:27,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:27,918 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501327917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,918 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:27,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:27,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40a6faeb[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:27,920 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:27,920 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:27,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501327919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:27,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501327919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,923 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,923 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:27,923 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:27,923 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:27,923 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,923 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:27,924 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,924 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:27,924 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:27,925 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:27,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:27,930 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:27,930 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:27,930 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:27,930 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:27,930 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:27,935 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,935 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,935 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:27,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:27,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:27,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:27,938 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,942 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403329ba25658784a3c811de340b308cdbf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403329ba25658784a3c811de340b308cdbf 2023-04-03 05:54:27,942 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:27,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:27,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:27,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:27,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:27,944 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:27,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:27,945 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:27,945 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:27,945 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,945 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:27,945 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:27,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:27,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:27,948 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:27,948 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:28,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:28,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:28,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501328020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501328022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:28,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501328023, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,048 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:28,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:28,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:28,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:28,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:28,052 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:28,052 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,052 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:28,052 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:28,053 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,053 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,057 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,058 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,058 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:28,058 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:28,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:28,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:28,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:28,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:28,060 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,061 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:28,061 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:28,061 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:28,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:28,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:28,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:28,070 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,070 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,070 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,070 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,072 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:28,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501328070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f05e30c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:28,072 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:28,072 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,075 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,075 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,075 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:28,075 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:28,076 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:28,081 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:28,081 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:28,081 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:28,081 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:28,081 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:28,081 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:28,083 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,083 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,083 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,084 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,089 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,089 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,089 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:28,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:28,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,089 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,089 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,089 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,089 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,089 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,089 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,089 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,090 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:28,090 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:28,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,190 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:28,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:28,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:28,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:28,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:28,193 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:28,193 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,193 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:28,193 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:28,194 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,194 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,196 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,196 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,196 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,196 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,197 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:28,197 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:28,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:28,200 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,200 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:28,200 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:28,200 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:28,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:28,202 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:28,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:28,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@725171c1[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:28,204 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:28,204 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,205 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:28,205 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,205 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:28,206 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,207 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:28,209 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:28,209 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:28,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:28,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:28,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:28,209 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:28,212 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,212 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,212 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,212 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,213 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,216 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,216 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:28,216 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:28,216 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:28,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,216 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,216 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,217 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,217 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,217 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,217 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,217 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:28,217 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:28,217 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:28,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501328225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,226 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:28,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501328225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:28,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501328226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,317 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:28,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:28,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:28,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:28,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:28,320 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:28,320 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,320 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:28,320 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:28,321 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,321 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,322 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,322 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,322 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,323 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:28,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:28,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:28,323 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:28,323 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:28,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:28,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:28,324 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:28,324 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:28,324 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,324 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:28,324 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:28,324 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,324 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:28,324 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:28,324 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,324 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:28,324 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:28,326 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,326 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,326 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,326 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:28,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ebbd06b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:28,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:28,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,330 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,330 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:28,330 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:28,330 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:28,330 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,330 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:28,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:28,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:28,334 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:28,334 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:28,334 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:28,334 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:28,334 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:28,335 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,335 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,335 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,336 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,339 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,339 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,339 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,339 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,339 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,340 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,340 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,340 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:28,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:28,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,340 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,340 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,341 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:28,341 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:28,354 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=608, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/638a31e33baa4c2e85299d0794c5cbdd 2023-04-03 05:54:28,441 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:28,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:28,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:28,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:28,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:28,444 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:28,444 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,444 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:28,444 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:28,444 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,445 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,448 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,448 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,448 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,448 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:28,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:28,450 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:28,450 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:28,450 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,450 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:28,450 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:28,450 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:28,451 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:28,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,452 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,453 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,453 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:28,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@393d1af5[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:28,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:28,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,458 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:28,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:28,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,460 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:28,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,461 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,461 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:28,463 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:28,463 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:28,463 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:28,463 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:28,463 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:28,463 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:28,464 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,464 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,465 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,465 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,467 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,469 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,469 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,469 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,469 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,469 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:28,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,469 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,472 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:28,472 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:28,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:28,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501328528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,530 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:28,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501328529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,532 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:28,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501328531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,572 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:28,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:28,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:28,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:28,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:28,575 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:28,575 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,575 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:28,575 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:28,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:28,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:28,578 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:28,578 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:28,579 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,579 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:28,579 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:28,579 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:28,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:28,580 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,580 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,581 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,581 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:28,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79e0835d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:28,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:28,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,583 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:28,584 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:28,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:28,586 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:28,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:28,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:28,588 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:28,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:28,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:28,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:28,588 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:28,590 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,590 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,590 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,590 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:28,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:28,594 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:28,594 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:28,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:28,695 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:28,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:28,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:28,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:28,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:28,698 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:28,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:28,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:28,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,701 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,701 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,701 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,701 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:28,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,701 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:28,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,702 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:28,702 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:28,703 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:28,703 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,703 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:28,703 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:28,703 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:28,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:28,705 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,705 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,706 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,706 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:28,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c63ad9[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:28,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:28,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:28,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:28,709 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,709 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:28,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:28,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:28,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,710 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:28,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:28,713 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:28,713 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:28,713 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:28,713 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:28,713 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:28,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,715 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,717 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,719 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:28,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:28,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:28,720 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:28,720 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:28,720 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,720 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,720 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,765 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/8afa22f3ee2c4c9299adbff12938389f 2023-04-03 05:54:28,820 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:28,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:28,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:28,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:28,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:28,823 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:28,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:28,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:28,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,824 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:28,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:28,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:28,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:28,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:28,826 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,826 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:28,826 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:28,826 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:28,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:28,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:28,828 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,829 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,829 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:28,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ee8a22[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:28,830 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:28,830 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,832 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,832 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:28,832 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:28,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:28,832 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:28,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:28,836 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:28,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:28,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:28,838 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:28,838 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:28,839 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:28,839 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:28,839 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:28,840 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,840 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,841 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,841 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,842 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,845 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:28,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:28,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:28,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,846 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:28,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,846 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:28,946 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:28,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:28,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:28,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:28,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:28,949 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:28,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:28,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:28,950 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,950 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,951 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,951 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,952 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,952 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:28,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:28,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:28,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:28,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:28,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:28,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:28,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:28,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,955 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:28,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41d5d078[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:28,956 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:28,957 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,958 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,958 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:28,958 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:28,958 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:28,958 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,958 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:28,961 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:28,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:28,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:28,963 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:28,964 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:28,964 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:28,964 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:28,964 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:28,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:28,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,966 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:28,966 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:28,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:28,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:28,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:28,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:28,970 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:28,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:28,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:28,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:28,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:28,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:28,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:28,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:28,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:28,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:28,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:28,971 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:28,971 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:29,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:29,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501329030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:29,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501329032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:29,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501329033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,071 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:29,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:29,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:29,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:29,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:29,074 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:29,074 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:29,074 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:29,074 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:29,075 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:29,075 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:29,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,076 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:29,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:29,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:29,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:29,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:29,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:29,077 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:29,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:29,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:29,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:29,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:29,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:29,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:29,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:29,078 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:29,078 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:29,078 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:29,078 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:29,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:29,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:29,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:29,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:29,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:29,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501329079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,081 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:29,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:29,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:29,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:29,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cd25974[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:29,083 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:29,083 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:29,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:29,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:29,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:29,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:29,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:29,084 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:29,084 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,084 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:29,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,085 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:29,086 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:29,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,086 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:29,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:29,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:29,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:29,088 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:29,088 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:29,088 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:29,088 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:29,089 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:29,090 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:29,090 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:29,090 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:29,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:29,091 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:29,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:29,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:29,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:29,092 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:29,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:29,096 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:29,096 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:29,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:29,096 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:29,096 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,096 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:29,096 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:29,096 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:29,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:29,096 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:29,096 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:29,096 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:29,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:29,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:29,096 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:29,096 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:29,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:29,180 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/25f6eac09c994cf886d35cbc41370b86 2023-04-03 05:54:29,184 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/638a31e33baa4c2e85299d0794c5cbdd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/638a31e33baa4c2e85299d0794c5cbdd 2023-04-03 05:54:29,190 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/638a31e33baa4c2e85299d0794c5cbdd, entries=200, sequenceid=608, filesize=32.3 K 2023-04-03 05:54:29,191 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/8afa22f3ee2c4c9299adbff12938389f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/8afa22f3ee2c4c9299adbff12938389f 2023-04-03 05:54:29,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/8afa22f3ee2c4c9299adbff12938389f, entries=150, sequenceid=608, filesize=11.9 K 2023-04-03 05:54:29,196 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/25f6eac09c994cf886d35cbc41370b86 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/25f6eac09c994cf886d35cbc41370b86 2023-04-03 05:54:29,197 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:29,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:29,199 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:29,199 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/25f6eac09c994cf886d35cbc41370b86, entries=150, sequenceid=608, filesize=11.9 K 2023-04-03 05:54:29,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:29,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:29,200 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:29,200 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:29,200 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:29,200 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:29,200 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 574b09e98d501def4bb65b12ef4de07a in 1299ms, sequenceid=608, compaction requested=true 2023-04-03 05:54:29,200 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:29,200 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:29,200 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:29,200 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:29,201 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:29,201 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:29,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:29,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:29,202 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,202 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:29,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:29,202 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:29,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:29,202 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:29,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:29,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:29,203 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:29,203 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:29,203 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:29,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:29,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:29,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:29,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:29,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:29,203 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:29,203 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:29,203 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:54:29,204 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:29,204 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:29,204 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:29,204 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:29,204 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:29,204 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:29,208 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:29,208 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:29,208 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/A is initiating minor compaction (all files) 2023-04-03 05:54:29,208 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/B is initiating minor compaction (all files) 2023-04-03 05:54:29,209 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/A in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:29,209 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/B in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:29,209 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/59b7952a08ef46c698a4e34efb89dad2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/1c4da99d702a45bfa14e14cd2e754071, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/638a31e33baa4c2e85299d0794c5cbdd] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=91.0 K 2023-04-03 05:54:29,209 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/de12715bb6284f05939a9d9e8c24d5f2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/44b6dac4d24a4d77822bda1e28cf7289, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/8afa22f3ee2c4c9299adbff12938389f] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=36.7 K 2023-04-03 05:54:29,209 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 59b7952a08ef46c698a4e34efb89dad2, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680501266378 2023-04-03 05:54:29,215 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting de12715bb6284f05939a9d9e8c24d5f2, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680501266378 2023-04-03 05:54:29,215 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 1c4da99d702a45bfa14e14cd2e754071, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1680501266717 2023-04-03 05:54:29,216 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 44b6dac4d24a4d77822bda1e28cf7289, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1680501266717 2023-04-03 05:54:29,216 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 638a31e33baa4c2e85299d0794c5cbdd, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1680501266934 2023-04-03 05:54:29,216 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 8afa22f3ee2c4c9299adbff12938389f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1680501266934 2023-04-03 05:54:29,225 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#B#compaction#219 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:29,249 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/e0bf6d03bf9a448ba2f11b52793b7e8a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/e0bf6d03bf9a448ba2f11b52793b7e8a 2023-04-03 05:54:29,250 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#A#compaction#220 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:29,256 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/B of 574b09e98d501def4bb65b12ef4de07a into e0bf6d03bf9a448ba2f11b52793b7e8a(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:29,256 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:29,256 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/B, priority=13, startTime=1680501269201; duration=0sec 2023-04-03 05:54:29,256 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:29,257 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:29,258 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:29,259 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/C is initiating minor compaction (all files) 2023-04-03 05:54:29,259 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/C in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:29,259 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/26ca58f6915d47eeac660ffa1d6a9969, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6db3c8eecc8b42368414e2c3611c0927, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/25f6eac09c994cf886d35cbc41370b86] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=36.7 K 2023-04-03 05:54:29,259 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 26ca58f6915d47eeac660ffa1d6a9969, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1680501266378 2023-04-03 05:54:29,260 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6db3c8eecc8b42368414e2c3611c0927, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=586, earliestPutTs=1680501266717 2023-04-03 05:54:29,260 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 25f6eac09c994cf886d35cbc41370b86, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1680501266934 2023-04-03 05:54:29,271 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#C#compaction#221 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:29,297 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/76ca9afb36ba414a9281861c7f1cce96 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/76ca9afb36ba414a9281861c7f1cce96 2023-04-03 05:54:29,302 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/C of 574b09e98d501def4bb65b12ef4de07a into 76ca9afb36ba414a9281861c7f1cce96(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:29,302 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:29,302 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/C, priority=13, startTime=1680501269201; duration=0sec 2023-04-03 05:54:29,302 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:29,648 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403d833ff8a45934773b501ae4f32a991a6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d833ff8a45934773b501ae4f32a991a6 2023-04-03 05:54:30,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:30,039 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:30,053 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=622, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/4dac56dcafce40f7bf3021682ac06753 2023-04-03 05:54:30,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501330059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501330060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,060 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501330059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,068 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=622 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/93522d9f6912459ead90cdb09c2bf698 2023-04-03 05:54:30,072 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/59a2f7ba47bc43658d2972845c434cc7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/59a2f7ba47bc43658d2972845c434cc7 2023-04-03 05:54:30,080 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/A of 574b09e98d501def4bb65b12ef4de07a into 59a2f7ba47bc43658d2972845c434cc7(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:30,080 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:30,080 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/A, priority=13, startTime=1680501269200; duration=0sec 2023-04-03 05:54:30,080 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:30,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501330162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501330162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501330164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501330366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501330366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,367 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501330367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,486 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=622 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/0de3582819a54565a29fc446f2fc128c 2023-04-03 05:54:30,491 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/4dac56dcafce40f7bf3021682ac06753 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4dac56dcafce40f7bf3021682ac06753 2023-04-03 05:54:30,495 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4dac56dcafce40f7bf3021682ac06753, entries=150, sequenceid=622, filesize=25.4 K 2023-04-03 05:54:30,496 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/93522d9f6912459ead90cdb09c2bf698 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/93522d9f6912459ead90cdb09c2bf698 2023-04-03 05:54:30,500 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/93522d9f6912459ead90cdb09c2bf698, entries=150, sequenceid=622, filesize=11.9 K 2023-04-03 05:54:30,501 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/0de3582819a54565a29fc446f2fc128c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/0de3582819a54565a29fc446f2fc128c 2023-04-03 05:54:30,506 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/0de3582819a54565a29fc446f2fc128c, entries=150, sequenceid=622, filesize=11.9 K 2023-04-03 05:54:30,507 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 574b09e98d501def4bb65b12ef4de07a in 1304ms, sequenceid=622, compaction requested=false 2023-04-03 05:54:30,507 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:30,507 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:30,507 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:30,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:30,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,514 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:30,514 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:30,514 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:30,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:30,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:30,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@df3bcc5[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:30,516 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:30,516 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:30,517 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:30,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:30,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:30,518 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:30,518 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:30,518 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:30,518 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,518 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:30,519 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:30,519 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:30,519 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:30,519 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:30,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:30,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:30,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:30,522 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:30,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:30,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:30,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:30,522 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:30,524 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:30,524 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:30,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:30,524 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:30,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:30,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:30,525 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:30,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:30,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:30,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:30,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,532 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:30,532 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:30,532 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:30,532 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:30,532 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:30,532 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:30,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:30,533 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:30,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:30,533 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,533 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:30,533 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:30,533 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:30,533 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:30,533 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:30,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:30,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:30,633 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:30,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:30,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:30,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:30,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:30,642 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:30,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:30,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:30,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:30,644 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:30,644 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:30,646 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:30,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:30,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:30,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:30,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:30,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:30,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:30,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:30,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:30,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:30,650 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:30,651 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:30,651 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:30,652 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:30,652 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:30,652 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:30,652 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. started... 2023-04-03 05:54:30,652 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 05:54:30,652 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:30,652 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:30,652 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:30,652 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:30,652 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:30,652 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:30,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:30,670 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. as already flushing 2023-04-03 05:54:30,685 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,686 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403dd47a8e8e08d4e6784652140a2c013cd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403dd47a8e8e08d4e6784652140a2c013cd 2023-04-03 05:54:30,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501330683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501330685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,688 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501330686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,723 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=644, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/4a2154d6060c458eac5ee8ddfd1dd210 2023-04-03 05:54:30,787 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501330787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,792 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501330789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501330790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501330992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501330993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:30,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:30,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501330996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:31,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53196 deadline: 1680501331089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,092 DEBUG [Thread-913] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:54:31,148 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/7308ab600f70473b90448e26b8a50fda 2023-04-03 05:54:31,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:31,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53200 deadline: 1680501331295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:31,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53226 deadline: 1680501331296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:31,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53244 deadline: 1680501331301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:31,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53234 deadline: 1680501331485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,489 DEBUG [Thread-917] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18233 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:54:31,558 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=644 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/467970e0a5544a5fa1d91c5b30c00feb 2023-04-03 05:54:31,562 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/4a2154d6060c458eac5ee8ddfd1dd210 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4a2154d6060c458eac5ee8ddfd1dd210 2023-04-03 05:54:31,566 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4a2154d6060c458eac5ee8ddfd1dd210, entries=150, sequenceid=644, filesize=25.4 K 2023-04-03 05:54:31,567 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/7308ab600f70473b90448e26b8a50fda as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/7308ab600f70473b90448e26b8a50fda 2023-04-03 05:54:31,571 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/7308ab600f70473b90448e26b8a50fda, entries=150, sequenceid=644, filesize=11.9 K 2023-04-03 05:54:31,572 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/467970e0a5544a5fa1d91c5b30c00feb as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/467970e0a5544a5fa1d91c5b30c00feb 2023-04-03 05:54:31,575 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/467970e0a5544a5fa1d91c5b30c00feb, entries=150, sequenceid=644, filesize=11.9 K 2023-04-03 05:54:31,576 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 574b09e98d501def4bb65b12ef4de07a in 924ms, sequenceid=644, compaction requested=true 2023-04-03 05:54:31,576 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:31,576 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:31,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:31,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:31,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:31,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:31,581 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:31,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:31,581 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:31,581 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:31,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:31,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:31,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:31,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@692898c3[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:31,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:31,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:31,584 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:31,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:31,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:31,584 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:31,584 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:31,584 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:31,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:31,586 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:31,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:31,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:31,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:31,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:31,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:31,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:31,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:31,588 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:31,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:31,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:31,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:31,588 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:31,590 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:31,590 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:31,590 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:31,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:31,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:31,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:31,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:31,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:31,595 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:31,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:31,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:31,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:31,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:31,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:31,595 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:31,595 DEBUG [Thread-923] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:31,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:31,595 DEBUG [Thread-923] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:31,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:31,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:31,596 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:31,596 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:31,596 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:31,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:31,682 DEBUG [Thread-930] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x41173aec to 127.0.0.1:63744 2023-04-03 05:54:31,682 DEBUG [Thread-930] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:31,683 DEBUG [Thread-926] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6b416c44 to 127.0.0.1:63744 2023-04-03 05:54:31,683 DEBUG [Thread-926] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:31,683 DEBUG [Thread-928] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x204b5880 to 127.0.0.1:63744 2023-04-03 05:54:31,683 DEBUG [Thread-928] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:31,684 DEBUG [Thread-924] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2668d41d to 127.0.0.1:63744 2023-04-03 05:54:31,684 DEBUG [Thread-924] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:31,696 DEBUG [Thread-923] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:31,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:31,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:31,801 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:54:31,802 DEBUG [Thread-921] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x40e0a3dc to 127.0.0.1:63744 2023-04-03 05:54:31,802 DEBUG [Thread-921] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:31,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:31,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:31,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:31,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:31,802 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:31,802 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:31,803 DEBUG [Thread-915] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52d98fed to 127.0.0.1:63744 2023-04-03 05:54:31,803 DEBUG [Thread-915] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:31,807 DEBUG [Thread-919] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x25441895 to 127.0.0.1:63744 2023-04-03 05:54:31,807 DEBUG [Thread-919] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:32,220 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040304315f0dba9e4d58a20d6ee81176a80c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040304315f0dba9e4d58a20d6ee81176a80c 2023-04-03 05:54:32,625 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=659, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/02bc86d9f9084736b2513eefae283250 2023-04-03 05:54:33,035 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/2e7bbe7f96f44e36aedbdf56b366f23d 2023-04-03 05:54:33,443 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=659 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/01a05a1155a649a5bd8d4646c942fdeb 2023-04-03 05:54:33,447 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/02bc86d9f9084736b2513eefae283250 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/02bc86d9f9084736b2513eefae283250 2023-04-03 05:54:33,450 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/02bc86d9f9084736b2513eefae283250, entries=150, sequenceid=659, filesize=25.4 K 2023-04-03 05:54:33,451 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/2e7bbe7f96f44e36aedbdf56b366f23d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/2e7bbe7f96f44e36aedbdf56b366f23d 2023-04-03 05:54:33,453 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/2e7bbe7f96f44e36aedbdf56b366f23d, entries=150, sequenceid=659, filesize=11.9 K 2023-04-03 05:54:33,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/01a05a1155a649a5bd8d4646c942fdeb as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/01a05a1155a649a5bd8d4646c942fdeb 2023-04-03 05:54:33,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/01a05a1155a649a5bd8d4646c942fdeb, entries=150, sequenceid=659, filesize=11.9 K 2023-04-03 05:54:33,458 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=6.71 KB/6870 for 574b09e98d501def4bb65b12ef4de07a in 1657ms, sequenceid=659, compaction requested=true 2023-04-03 05:54:33,458 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:33,458 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:33,458 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:33,458 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:33,458 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:33,458 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:33,459 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105274 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:33,459 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:33,459 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/A is initiating minor compaction (all files) 2023-04-03 05:54:33,459 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/B is initiating minor compaction (all files) 2023-04-03 05:54:33,459 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/A in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:33,459 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/B in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:33,459 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/59a2f7ba47bc43658d2972845c434cc7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4dac56dcafce40f7bf3021682ac06753, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4a2154d6060c458eac5ee8ddfd1dd210, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/02bc86d9f9084736b2513eefae283250] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=102.8 K 2023-04-03 05:54:33,459 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/e0bf6d03bf9a448ba2f11b52793b7e8a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/93522d9f6912459ead90cdb09c2bf698, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/7308ab600f70473b90448e26b8a50fda, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/2e7bbe7f96f44e36aedbdf56b366f23d] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=48.7 K 2023-04-03 05:54:33,459 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 59a2f7ba47bc43658d2972845c434cc7, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1680501266934 2023-04-03 05:54:33,459 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e0bf6d03bf9a448ba2f11b52793b7e8a, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1680501266934 2023-04-03 05:54:33,459 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 4dac56dcafce40f7bf3021682ac06753, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1680501267913 2023-04-03 05:54:33,460 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 93522d9f6912459ead90cdb09c2bf698, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1680501267913 2023-04-03 05:54:33,460 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 4a2154d6060c458eac5ee8ddfd1dd210, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1680501270056 2023-04-03 05:54:33,460 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 7308ab600f70473b90448e26b8a50fda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1680501270056 2023-04-03 05:54:33,460 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 02bc86d9f9084736b2513eefae283250, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1680501270679 2023-04-03 05:54:33,460 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 2e7bbe7f96f44e36aedbdf56b366f23d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1680501270679 2023-04-03 05:54:33,466 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#B#compaction#230 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:33,469 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#A#compaction#231 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:33,474 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/c6be9300b2174cb7a1b98abff764191a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/c6be9300b2174cb7a1b98abff764191a 2023-04-03 05:54:33,478 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/B of 574b09e98d501def4bb65b12ef4de07a into c6be9300b2174cb7a1b98abff764191a(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:33,478 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:33,478 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/B, priority=12, startTime=1680501273458; duration=0sec 2023-04-03 05:54:33,478 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:33,478 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:33,479 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:33,479 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 574b09e98d501def4bb65b12ef4de07a/C is initiating minor compaction (all files) 2023-04-03 05:54:33,479 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 574b09e98d501def4bb65b12ef4de07a/C in TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:33,479 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/76ca9afb36ba414a9281861c7f1cce96, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/0de3582819a54565a29fc446f2fc128c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/467970e0a5544a5fa1d91c5b30c00feb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/01a05a1155a649a5bd8d4646c942fdeb] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp, totalSize=48.7 K 2023-04-03 05:54:33,480 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 76ca9afb36ba414a9281861c7f1cce96, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=608, earliestPutTs=1680501266934 2023-04-03 05:54:33,480 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0de3582819a54565a29fc446f2fc128c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=622, earliestPutTs=1680501267913 2023-04-03 05:54:33,480 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 467970e0a5544a5fa1d91c5b30c00feb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=644, earliestPutTs=1680501270056 2023-04-03 05:54:33,481 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 01a05a1155a649a5bd8d4646c942fdeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=659, earliestPutTs=1680501270679 2023-04-03 05:54:33,493 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 574b09e98d501def4bb65b12ef4de07a#C#compaction#232 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:33,900 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/770b1599a71d4adaa7bb81dc2c9f9c29 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/770b1599a71d4adaa7bb81dc2c9f9c29 2023-04-03 05:54:33,904 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/C of 574b09e98d501def4bb65b12ef4de07a into 770b1599a71d4adaa7bb81dc2c9f9c29(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:33,904 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:33,904 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/C, priority=12, startTime=1680501273458; duration=0sec 2023-04-03 05:54:33,905 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:34,698 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/18e489d2159b453590e9182a5c387359 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/18e489d2159b453590e9182a5c387359 2023-04-03 05:54:34,702 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 574b09e98d501def4bb65b12ef4de07a/A of 574b09e98d501def4bb65b12ef4de07a into 18e489d2159b453590e9182a5c387359(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-03 05:54:34,702 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:34,702 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a., storeName=574b09e98d501def4bb65b12ef4de07a/A, priority=12, startTime=1680501273458; duration=1sec 2023-04-03 05:54:34,702 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:35,104 DEBUG [Thread-913] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63ebf1e9 to 127.0.0.1:63744 2023-04-03 05:54:35,104 DEBUG [Thread-913] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:35,897 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-03 05:54:41,590 DEBUG [Thread-917] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x18496e17 to 127.0.0.1:63744 2023-04-03 05:54:41,590 DEBUG [Thread-917] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:41,590 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 05:54:41,590 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 154 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 119 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 39 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 86 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 150 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 5544 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 5113 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 2244 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 6732 rows 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 2250 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 6750 rows 2023-04-03 05:54:41,591 INFO [Listener at localhost/37275] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 05:54:41,591 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ab850e7 to 127.0.0.1:63744 2023-04-03 05:54:41,591 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:54:41,593 INFO [Listener at localhost/37275] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 05:54:41,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-03 05:54:41,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:41,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 05:54:41,597 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501281597"}]},"ts":"1680501281597"} 2023-04-03 05:54:41,598 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 05:54:41,602 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 05:54:41,602 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=574b09e98d501def4bb65b12ef4de07a, UNASSIGN}] 2023-04-03 05:54:41,604 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=574b09e98d501def4bb65b12ef4de07a, UNASSIGN 2023-04-03 05:54:41,604 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=574b09e98d501def4bb65b12ef4de07a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:41,604 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501281604"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501281604"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501281604"}]},"ts":"1680501281604"} 2023-04-03 05:54:41,605 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:54:41,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 05:54:41,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:41,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 574b09e98d501def4bb65b12ef4de07a, disabling compactions & flushes 2023-04-03 05:54:41,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:41,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:41,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. after waiting 0 ms 2023-04-03 05:54:41,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:41,757 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 574b09e98d501def4bb65b12ef4de07a 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-03 05:54:41,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=A 2023-04-03 05:54:41,757 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:41,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=B 2023-04-03 05:54:41,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:41,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 574b09e98d501def4bb65b12ef4de07a, store=C 2023-04-03 05:54:41,758 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:41,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 05:54:42,171 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304036a4f464e00ed4a18a26c2638c090d0aa to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036a4f464e00ed4a18a26c2638c090d0aa 2023-04-03 05:54:42,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 05:54:42,576 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=668, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/10c051c5e945436a98bdb96f947dc25b 2023-04-03 05:54:42,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 05:54:42,985 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/b1630ceb41c94278b67b053c98f100a8 2023-04-03 05:54:43,393 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=668 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/203f940bc4ca492f884fdb2814b8a6fe 2023-04-03 05:54:43,398 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/A/10c051c5e945436a98bdb96f947dc25b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/10c051c5e945436a98bdb96f947dc25b 2023-04-03 05:54:43,402 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/10c051c5e945436a98bdb96f947dc25b, entries=150, sequenceid=668, filesize=25.4 K 2023-04-03 05:54:43,403 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/B/b1630ceb41c94278b67b053c98f100a8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/b1630ceb41c94278b67b053c98f100a8 2023-04-03 05:54:43,407 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/b1630ceb41c94278b67b053c98f100a8, entries=150, sequenceid=668, filesize=11.9 K 2023-04-03 05:54:43,408 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/.tmp/C/203f940bc4ca492f884fdb2814b8a6fe as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/203f940bc4ca492f884fdb2814b8a6fe 2023-04-03 05:54:43,412 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/203f940bc4ca492f884fdb2814b8a6fe, entries=150, sequenceid=668, filesize=11.9 K 2023-04-03 05:54:43,413 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 574b09e98d501def4bb65b12ef4de07a in 1656ms, sequenceid=668, compaction requested=false 2023-04-03 05:54:43,436 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/09304f0180b344e6adf748bd0f9f1ee7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/f3211418e47e4182ade9ade3914f787c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/8c90e4cec55c4aaeaf70bc910cef8ecf, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/cb68b13b3d42464f873b5e481f8dfb51, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/c774e64471cb45b2b4456053ea355cb2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/7991607bb4894adeb98cf9133c4367d0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/2f36ca9b1278489283ef671bc4198045, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/b5595f86b1374f70b2e36b00405b6fc5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/b9fe9df10cf04a01981686ada3454325, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4d56ed6499e243e38a0ce3d159a72441, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/35e404745f5d4eaf9c4eb8ef906e7f40, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/925a044853ca4661944904569f2a4ae4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/533b23c44f2f46cfa31ae5f0b176f9b2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/2fe03fd0f5e9425bb4e57d70c4a7e00d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/12272198cc8740bb9b409d9147b8656e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/ec0fa778724645bdbe09b20545f48eea, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/40182c28094249e49e26a7b020d2cb5b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/63c5685a1e2e4d469be85bc9833f8280, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/02cc96ae79bb4345be0ad8426bd85cfa, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/88269a9e925c413fa1765da8e9b3cee9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/e08857f97e674e8d815ed38654c73b0f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/7e8f7c4494bb41849c528fc5350d70c5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/23773312f3594ab3883910637c1397fe, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/5abcdcfcb7de4fc2a180c49a24683176, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/395e3e4d54fe4d42a8329ebcc03e7d65, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/0847547187e84db39c7d34ab0afd7b56, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/0afb273c51b94fefac5df55a10fad33c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/225c06f31a12416d88c3aa64cd7b5415, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/cf8e2fa2c9b64bf7ab39552fe397041d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/59b7952a08ef46c698a4e34efb89dad2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/1c4da99d702a45bfa14e14cd2e754071, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/638a31e33baa4c2e85299d0794c5cbdd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/59a2f7ba47bc43658d2972845c434cc7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4dac56dcafce40f7bf3021682ac06753, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4a2154d6060c458eac5ee8ddfd1dd210, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/02bc86d9f9084736b2513eefae283250] to archive 2023-04-03 05:54:43,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:54:43,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/09304f0180b344e6adf748bd0f9f1ee7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/09304f0180b344e6adf748bd0f9f1ee7 2023-04-03 05:54:43,440 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/f3211418e47e4182ade9ade3914f787c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/f3211418e47e4182ade9ade3914f787c 2023-04-03 05:54:43,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/8c90e4cec55c4aaeaf70bc910cef8ecf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/8c90e4cec55c4aaeaf70bc910cef8ecf 2023-04-03 05:54:43,442 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/cb68b13b3d42464f873b5e481f8dfb51 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/cb68b13b3d42464f873b5e481f8dfb51 2023-04-03 05:54:43,443 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/c774e64471cb45b2b4456053ea355cb2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/c774e64471cb45b2b4456053ea355cb2 2023-04-03 05:54:43,444 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/7991607bb4894adeb98cf9133c4367d0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/7991607bb4894adeb98cf9133c4367d0 2023-04-03 05:54:43,445 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/2f36ca9b1278489283ef671bc4198045 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/2f36ca9b1278489283ef671bc4198045 2023-04-03 05:54:43,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/b5595f86b1374f70b2e36b00405b6fc5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/b5595f86b1374f70b2e36b00405b6fc5 2023-04-03 05:54:43,447 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/b9fe9df10cf04a01981686ada3454325 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/b9fe9df10cf04a01981686ada3454325 2023-04-03 05:54:43,449 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4d56ed6499e243e38a0ce3d159a72441 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4d56ed6499e243e38a0ce3d159a72441 2023-04-03 05:54:43,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/35e404745f5d4eaf9c4eb8ef906e7f40 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/35e404745f5d4eaf9c4eb8ef906e7f40 2023-04-03 05:54:43,451 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/925a044853ca4661944904569f2a4ae4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/925a044853ca4661944904569f2a4ae4 2023-04-03 05:54:43,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/533b23c44f2f46cfa31ae5f0b176f9b2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/533b23c44f2f46cfa31ae5f0b176f9b2 2023-04-03 05:54:43,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/2fe03fd0f5e9425bb4e57d70c4a7e00d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/2fe03fd0f5e9425bb4e57d70c4a7e00d 2023-04-03 05:54:43,454 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/12272198cc8740bb9b409d9147b8656e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/12272198cc8740bb9b409d9147b8656e 2023-04-03 05:54:43,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/ec0fa778724645bdbe09b20545f48eea to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/ec0fa778724645bdbe09b20545f48eea 2023-04-03 05:54:43,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/40182c28094249e49e26a7b020d2cb5b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/40182c28094249e49e26a7b020d2cb5b 2023-04-03 05:54:43,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/63c5685a1e2e4d469be85bc9833f8280 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/63c5685a1e2e4d469be85bc9833f8280 2023-04-03 05:54:43,458 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/02cc96ae79bb4345be0ad8426bd85cfa to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/02cc96ae79bb4345be0ad8426bd85cfa 2023-04-03 05:54:43,459 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/88269a9e925c413fa1765da8e9b3cee9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/88269a9e925c413fa1765da8e9b3cee9 2023-04-03 05:54:43,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/e08857f97e674e8d815ed38654c73b0f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/e08857f97e674e8d815ed38654c73b0f 2023-04-03 05:54:43,461 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/7e8f7c4494bb41849c528fc5350d70c5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/7e8f7c4494bb41849c528fc5350d70c5 2023-04-03 05:54:43,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/23773312f3594ab3883910637c1397fe to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/23773312f3594ab3883910637c1397fe 2023-04-03 05:54:43,463 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/5abcdcfcb7de4fc2a180c49a24683176 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/5abcdcfcb7de4fc2a180c49a24683176 2023-04-03 05:54:43,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/395e3e4d54fe4d42a8329ebcc03e7d65 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/395e3e4d54fe4d42a8329ebcc03e7d65 2023-04-03 05:54:43,465 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/0847547187e84db39c7d34ab0afd7b56 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/0847547187e84db39c7d34ab0afd7b56 2023-04-03 05:54:43,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/0afb273c51b94fefac5df55a10fad33c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/0afb273c51b94fefac5df55a10fad33c 2023-04-03 05:54:43,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/225c06f31a12416d88c3aa64cd7b5415 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/225c06f31a12416d88c3aa64cd7b5415 2023-04-03 05:54:43,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/cf8e2fa2c9b64bf7ab39552fe397041d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/cf8e2fa2c9b64bf7ab39552fe397041d 2023-04-03 05:54:43,469 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/59b7952a08ef46c698a4e34efb89dad2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/59b7952a08ef46c698a4e34efb89dad2 2023-04-03 05:54:43,470 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/1c4da99d702a45bfa14e14cd2e754071 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/1c4da99d702a45bfa14e14cd2e754071 2023-04-03 05:54:43,471 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/638a31e33baa4c2e85299d0794c5cbdd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/638a31e33baa4c2e85299d0794c5cbdd 2023-04-03 05:54:43,472 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/59a2f7ba47bc43658d2972845c434cc7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/59a2f7ba47bc43658d2972845c434cc7 2023-04-03 05:54:43,474 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4dac56dcafce40f7bf3021682ac06753 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4dac56dcafce40f7bf3021682ac06753 2023-04-03 05:54:43,475 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4a2154d6060c458eac5ee8ddfd1dd210 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/4a2154d6060c458eac5ee8ddfd1dd210 2023-04-03 05:54:43,476 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/02bc86d9f9084736b2513eefae283250 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/02bc86d9f9084736b2513eefae283250 2023-04-03 05:54:43,480 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/53974844e3b04ea8ab99b3fdea024195, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0e8cd1f5284546b58912fd137a654382, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/98e3b72ab2e14cd1843762077c3416cf, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/b88813d13e364bfbafb095650ada1322, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/a6fa374e48224681ab06405acd468114, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/f16460db815e4f5189e873aa56655ac8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/a7aa188159074109882955611e9f9717, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/cbb5ca7dfe29464f88d826b0399bf7c5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0f679bef98a746de8cc701df679af2f8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/09ce6768c0d744329a39744a28c16837, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/40ee4ab0a6954716ae520a4cd68f3878, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/915b354bd1264c65b21f48dc851640db, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/7e634fb8d35e4be391e67b6c26e26033, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/8b43f88f49674993bfa722c1240f676c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0337a461fb8b4b69b26e911f59989be0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/fa0cc3f9d23146789a9ad27a5c581f95, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/c38dea87be0448f99faa9cc02ed14aff, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/395118226dca4c2b97385cdd755eac12, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/ae38866e270443cea646fe8fe8afe911, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/ca028b3d0eec450cb2d992fdba2ec433, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/49edc184a0fb49e0a4a5f2887a5e89fc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/485e7ecf40d141209f5f3b7bacb95676, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/19d03acb357b436bbf02be3ddc522c35, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/12896c09ca1f4a9ba3f711f9bf72bd3e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0f9f8f874a7547de864db1575b91c5d1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/d26dfdbd41734c418cc7233de3fec63d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/cbb8599d4c9d43bf9751343b6f809b70, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/01b2402fc8ae44dfa8be36272d13b052, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/3518ae4b7d994ee8bd2c8b02c37eaaea, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/de12715bb6284f05939a9d9e8c24d5f2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/bbb20a22a0b24f1dbf1b7a59c23b094f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/44b6dac4d24a4d77822bda1e28cf7289, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/e0bf6d03bf9a448ba2f11b52793b7e8a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/8afa22f3ee2c4c9299adbff12938389f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/93522d9f6912459ead90cdb09c2bf698, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/7308ab600f70473b90448e26b8a50fda, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/2e7bbe7f96f44e36aedbdf56b366f23d] to archive 2023-04-03 05:54:43,481 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:54:43,482 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/53974844e3b04ea8ab99b3fdea024195 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/53974844e3b04ea8ab99b3fdea024195 2023-04-03 05:54:43,483 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0e8cd1f5284546b58912fd137a654382 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0e8cd1f5284546b58912fd137a654382 2023-04-03 05:54:43,484 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/98e3b72ab2e14cd1843762077c3416cf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/98e3b72ab2e14cd1843762077c3416cf 2023-04-03 05:54:43,485 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/b88813d13e364bfbafb095650ada1322 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/b88813d13e364bfbafb095650ada1322 2023-04-03 05:54:43,486 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/a6fa374e48224681ab06405acd468114 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/a6fa374e48224681ab06405acd468114 2023-04-03 05:54:43,487 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/f16460db815e4f5189e873aa56655ac8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/f16460db815e4f5189e873aa56655ac8 2023-04-03 05:54:43,488 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/a7aa188159074109882955611e9f9717 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/a7aa188159074109882955611e9f9717 2023-04-03 05:54:43,489 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/cbb5ca7dfe29464f88d826b0399bf7c5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/cbb5ca7dfe29464f88d826b0399bf7c5 2023-04-03 05:54:43,490 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0f679bef98a746de8cc701df679af2f8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0f679bef98a746de8cc701df679af2f8 2023-04-03 05:54:43,491 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/09ce6768c0d744329a39744a28c16837 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/09ce6768c0d744329a39744a28c16837 2023-04-03 05:54:43,492 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/40ee4ab0a6954716ae520a4cd68f3878 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/40ee4ab0a6954716ae520a4cd68f3878 2023-04-03 05:54:43,493 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/915b354bd1264c65b21f48dc851640db to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/915b354bd1264c65b21f48dc851640db 2023-04-03 05:54:43,494 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/7e634fb8d35e4be391e67b6c26e26033 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/7e634fb8d35e4be391e67b6c26e26033 2023-04-03 05:54:43,495 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/8b43f88f49674993bfa722c1240f676c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/8b43f88f49674993bfa722c1240f676c 2023-04-03 05:54:43,495 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0337a461fb8b4b69b26e911f59989be0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0337a461fb8b4b69b26e911f59989be0 2023-04-03 05:54:43,496 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/fa0cc3f9d23146789a9ad27a5c581f95 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/fa0cc3f9d23146789a9ad27a5c581f95 2023-04-03 05:54:43,497 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/c38dea87be0448f99faa9cc02ed14aff to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/c38dea87be0448f99faa9cc02ed14aff 2023-04-03 05:54:43,498 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/395118226dca4c2b97385cdd755eac12 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/395118226dca4c2b97385cdd755eac12 2023-04-03 05:54:43,499 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/ae38866e270443cea646fe8fe8afe911 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/ae38866e270443cea646fe8fe8afe911 2023-04-03 05:54:43,500 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/ca028b3d0eec450cb2d992fdba2ec433 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/ca028b3d0eec450cb2d992fdba2ec433 2023-04-03 05:54:43,501 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/49edc184a0fb49e0a4a5f2887a5e89fc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/49edc184a0fb49e0a4a5f2887a5e89fc 2023-04-03 05:54:43,502 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/485e7ecf40d141209f5f3b7bacb95676 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/485e7ecf40d141209f5f3b7bacb95676 2023-04-03 05:54:43,503 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/19d03acb357b436bbf02be3ddc522c35 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/19d03acb357b436bbf02be3ddc522c35 2023-04-03 05:54:43,504 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/12896c09ca1f4a9ba3f711f9bf72bd3e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/12896c09ca1f4a9ba3f711f9bf72bd3e 2023-04-03 05:54:43,505 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0f9f8f874a7547de864db1575b91c5d1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/0f9f8f874a7547de864db1575b91c5d1 2023-04-03 05:54:43,506 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/d26dfdbd41734c418cc7233de3fec63d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/d26dfdbd41734c418cc7233de3fec63d 2023-04-03 05:54:43,507 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/cbb8599d4c9d43bf9751343b6f809b70 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/cbb8599d4c9d43bf9751343b6f809b70 2023-04-03 05:54:43,508 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/01b2402fc8ae44dfa8be36272d13b052 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/01b2402fc8ae44dfa8be36272d13b052 2023-04-03 05:54:43,509 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/3518ae4b7d994ee8bd2c8b02c37eaaea to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/3518ae4b7d994ee8bd2c8b02c37eaaea 2023-04-03 05:54:43,510 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/de12715bb6284f05939a9d9e8c24d5f2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/de12715bb6284f05939a9d9e8c24d5f2 2023-04-03 05:54:43,511 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/bbb20a22a0b24f1dbf1b7a59c23b094f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/bbb20a22a0b24f1dbf1b7a59c23b094f 2023-04-03 05:54:43,512 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/44b6dac4d24a4d77822bda1e28cf7289 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/44b6dac4d24a4d77822bda1e28cf7289 2023-04-03 05:54:43,513 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/e0bf6d03bf9a448ba2f11b52793b7e8a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/e0bf6d03bf9a448ba2f11b52793b7e8a 2023-04-03 05:54:43,514 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/8afa22f3ee2c4c9299adbff12938389f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/8afa22f3ee2c4c9299adbff12938389f 2023-04-03 05:54:43,515 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/93522d9f6912459ead90cdb09c2bf698 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/93522d9f6912459ead90cdb09c2bf698 2023-04-03 05:54:43,517 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/7308ab600f70473b90448e26b8a50fda to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/7308ab600f70473b90448e26b8a50fda 2023-04-03 05:54:43,518 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/2e7bbe7f96f44e36aedbdf56b366f23d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/2e7bbe7f96f44e36aedbdf56b366f23d 2023-04-03 05:54:43,521 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/ea39e43cfb1c432bac760ad659b27a89, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6e5e0116849d4f1fa51a78fa8cb1d9b6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/05b7a3e562c84782b3d390ebf4b6511b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/8fd29ddc81474b279e4ba6e70fd606fe, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/81238b0c4b214aa7956806660318d0da, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/d284eee7f26d44a3ae5e77e29f4eaa18, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/76c33087cd424a3f88e7f87642ea6a91, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f630ed6d722d45ca8add323c9bef0f81, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/84f688ead23e47ae823c3eb637ec9932, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/aee339d8d1224ae28eb387bd3cea28a1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cd5eaaf40e1a4947881ef3c490506cfa, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/b9ce69f1ca6049ce80ed191035b39c7b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/1946cd0708044022a3f0309e631f89f0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/44aa33264db04119807a4ee52108a4bd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/700e20937e2242b3a1a0dac184b33898, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/2812a5fff78049fe87168a467d5ae6b9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/ac11d0f368c74c50b8197dc5dc443404, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/94194a4eb38f4c58bf9335f235e5e64d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/b70ff2c1da6d45118384da6e822e63e4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/91a36583f0164fabbb7648bb022209d1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cea57392e93e4081ab12f4167a6c0d11, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f3f013fe4553457fa910a1c52e32a165, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/811315447c3e43df8b2bc766a4a547bc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cdaf28ce082d4dd082d95ab903010bec, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6af2296bc51343b39fea3048ab83c82a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/548c915bf0aa4f51856654a5a177d34a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/2191cb783f9148db9790e8053d680a28, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/c917b830219e486eb939dc9ebe141385, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/3367c1294bb6434aba67b54f2ac8b5ea, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/26ca58f6915d47eeac660ffa1d6a9969, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f64fb35fb9c246e8ac660e1416ab36f7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6db3c8eecc8b42368414e2c3611c0927, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/76ca9afb36ba414a9281861c7f1cce96, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/25f6eac09c994cf886d35cbc41370b86, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/0de3582819a54565a29fc446f2fc128c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/467970e0a5544a5fa1d91c5b30c00feb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/01a05a1155a649a5bd8d4646c942fdeb] to archive 2023-04-03 05:54:43,522 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:54:43,524 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/ea39e43cfb1c432bac760ad659b27a89 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/ea39e43cfb1c432bac760ad659b27a89 2023-04-03 05:54:43,525 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6e5e0116849d4f1fa51a78fa8cb1d9b6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6e5e0116849d4f1fa51a78fa8cb1d9b6 2023-04-03 05:54:43,527 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/05b7a3e562c84782b3d390ebf4b6511b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/05b7a3e562c84782b3d390ebf4b6511b 2023-04-03 05:54:43,528 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/8fd29ddc81474b279e4ba6e70fd606fe to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/8fd29ddc81474b279e4ba6e70fd606fe 2023-04-03 05:54:43,529 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/81238b0c4b214aa7956806660318d0da to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/81238b0c4b214aa7956806660318d0da 2023-04-03 05:54:43,531 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/d284eee7f26d44a3ae5e77e29f4eaa18 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/d284eee7f26d44a3ae5e77e29f4eaa18 2023-04-03 05:54:43,532 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/76c33087cd424a3f88e7f87642ea6a91 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/76c33087cd424a3f88e7f87642ea6a91 2023-04-03 05:54:43,533 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f630ed6d722d45ca8add323c9bef0f81 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f630ed6d722d45ca8add323c9bef0f81 2023-04-03 05:54:43,535 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/84f688ead23e47ae823c3eb637ec9932 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/84f688ead23e47ae823c3eb637ec9932 2023-04-03 05:54:43,536 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/aee339d8d1224ae28eb387bd3cea28a1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/aee339d8d1224ae28eb387bd3cea28a1 2023-04-03 05:54:43,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cd5eaaf40e1a4947881ef3c490506cfa to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cd5eaaf40e1a4947881ef3c490506cfa 2023-04-03 05:54:43,537 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/b9ce69f1ca6049ce80ed191035b39c7b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/b9ce69f1ca6049ce80ed191035b39c7b 2023-04-03 05:54:43,538 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/1946cd0708044022a3f0309e631f89f0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/1946cd0708044022a3f0309e631f89f0 2023-04-03 05:54:43,539 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/44aa33264db04119807a4ee52108a4bd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/44aa33264db04119807a4ee52108a4bd 2023-04-03 05:54:43,540 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/700e20937e2242b3a1a0dac184b33898 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/700e20937e2242b3a1a0dac184b33898 2023-04-03 05:54:43,541 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/2812a5fff78049fe87168a467d5ae6b9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/2812a5fff78049fe87168a467d5ae6b9 2023-04-03 05:54:43,542 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/ac11d0f368c74c50b8197dc5dc443404 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/ac11d0f368c74c50b8197dc5dc443404 2023-04-03 05:54:43,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/94194a4eb38f4c58bf9335f235e5e64d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/94194a4eb38f4c58bf9335f235e5e64d 2023-04-03 05:54:43,543 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/b70ff2c1da6d45118384da6e822e63e4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/b70ff2c1da6d45118384da6e822e63e4 2023-04-03 05:54:43,544 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/91a36583f0164fabbb7648bb022209d1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/91a36583f0164fabbb7648bb022209d1 2023-04-03 05:54:43,545 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cea57392e93e4081ab12f4167a6c0d11 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cea57392e93e4081ab12f4167a6c0d11 2023-04-03 05:54:43,546 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f3f013fe4553457fa910a1c52e32a165 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f3f013fe4553457fa910a1c52e32a165 2023-04-03 05:54:43,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/811315447c3e43df8b2bc766a4a547bc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/811315447c3e43df8b2bc766a4a547bc 2023-04-03 05:54:43,547 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cdaf28ce082d4dd082d95ab903010bec to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/cdaf28ce082d4dd082d95ab903010bec 2023-04-03 05:54:43,548 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6af2296bc51343b39fea3048ab83c82a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6af2296bc51343b39fea3048ab83c82a 2023-04-03 05:54:43,549 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/548c915bf0aa4f51856654a5a177d34a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/548c915bf0aa4f51856654a5a177d34a 2023-04-03 05:54:43,550 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/2191cb783f9148db9790e8053d680a28 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/2191cb783f9148db9790e8053d680a28 2023-04-03 05:54:43,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/c917b830219e486eb939dc9ebe141385 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/c917b830219e486eb939dc9ebe141385 2023-04-03 05:54:43,551 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/3367c1294bb6434aba67b54f2ac8b5ea to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/3367c1294bb6434aba67b54f2ac8b5ea 2023-04-03 05:54:43,552 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/26ca58f6915d47eeac660ffa1d6a9969 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/26ca58f6915d47eeac660ffa1d6a9969 2023-04-03 05:54:43,553 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f64fb35fb9c246e8ac660e1416ab36f7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/f64fb35fb9c246e8ac660e1416ab36f7 2023-04-03 05:54:43,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6db3c8eecc8b42368414e2c3611c0927 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/6db3c8eecc8b42368414e2c3611c0927 2023-04-03 05:54:43,554 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/76ca9afb36ba414a9281861c7f1cce96 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/76ca9afb36ba414a9281861c7f1cce96 2023-04-03 05:54:43,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/25f6eac09c994cf886d35cbc41370b86 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/25f6eac09c994cf886d35cbc41370b86 2023-04-03 05:54:43,556 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/0de3582819a54565a29fc446f2fc128c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/0de3582819a54565a29fc446f2fc128c 2023-04-03 05:54:43,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/467970e0a5544a5fa1d91c5b30c00feb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/467970e0a5544a5fa1d91c5b30c00feb 2023-04-03 05:54:43,558 DEBUG [StoreCloser-TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/01a05a1155a649a5bd8d4646c942fdeb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/01a05a1155a649a5bd8d4646c942fdeb 2023-04-03 05:54:43,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/recovered.edits/671.seqid, newMaxSeqId=671, maxSeqId=4 2023-04-03 05:54:43,563 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a. 2023-04-03 05:54:43,563 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 574b09e98d501def4bb65b12ef4de07a: 2023-04-03 05:54:43,564 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:43,565 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=574b09e98d501def4bb65b12ef4de07a, regionState=CLOSED 2023-04-03 05:54:43,565 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501283565"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501283565"}]},"ts":"1680501283565"} 2023-04-03 05:54:43,567 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-03 05:54:43,567 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 574b09e98d501def4bb65b12ef4de07a, server=jenkins-hbase4.apache.org,37397,1680501214665 in 1.9610 sec 2023-04-03 05:54:43,568 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-03 05:54:43,568 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=574b09e98d501def4bb65b12ef4de07a, UNASSIGN in 1.9650 sec 2023-04-03 05:54:43,569 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501283568"}]},"ts":"1680501283568"} 2023-04-03 05:54:43,569 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 05:54:43,571 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 05:54:43,572 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9780 sec 2023-04-03 05:54:43,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-03 05:54:43,701 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-03 05:54:43,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-03 05:54:43,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:43,703 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:43,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-03 05:54:43,704 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:43,709 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:43,711 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/recovered.edits] 2023-04-03 05:54:43,713 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/10c051c5e945436a98bdb96f947dc25b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/10c051c5e945436a98bdb96f947dc25b 2023-04-03 05:54:43,714 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/18e489d2159b453590e9182a5c387359 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/A/18e489d2159b453590e9182a5c387359 2023-04-03 05:54:43,716 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/b1630ceb41c94278b67b053c98f100a8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/b1630ceb41c94278b67b053c98f100a8 2023-04-03 05:54:43,718 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/c6be9300b2174cb7a1b98abff764191a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/B/c6be9300b2174cb7a1b98abff764191a 2023-04-03 05:54:43,720 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/203f940bc4ca492f884fdb2814b8a6fe to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/203f940bc4ca492f884fdb2814b8a6fe 2023-04-03 05:54:43,721 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/770b1599a71d4adaa7bb81dc2c9f9c29 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/C/770b1599a71d4adaa7bb81dc2c9f9c29 2023-04-03 05:54:43,723 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/recovered.edits/671.seqid to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a/recovered.edits/671.seqid 2023-04-03 05:54:43,724 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/574b09e98d501def4bb65b12ef4de07a 2023-04-03 05:54:43,724 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 05:54:43,724 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 05:54:43,725 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-03 05:54:43,728 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040302c010e4698d49a98ca5c7faa97b2a89 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040302c010e4698d49a98ca5c7faa97b2a89 2023-04-03 05:54:43,729 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040304315f0dba9e4d58a20d6ee81176a80c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040304315f0dba9e4d58a20d6ee81176a80c 2023-04-03 05:54:43,730 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403053baaed87474fa9b18986be9732f19b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403053baaed87474fa9b18986be9732f19b 2023-04-03 05:54:43,731 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040309a87584302b4676b570be11be2c1a71 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040309a87584302b4676b570be11be2c1a71 2023-04-03 05:54:43,732 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030cfabe56ba2544c0a88522055ea2e846 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030cfabe56ba2544c0a88522055ea2e846 2023-04-03 05:54:43,733 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030e636430e4ef46a1b893870ff88d7419 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304030e636430e4ef46a1b893870ff88d7419 2023-04-03 05:54:43,735 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031104d04919c34e9fb22dda4c3732b5b4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031104d04919c34e9fb22dda4c3732b5b4 2023-04-03 05:54:43,736 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032867b16cbae841e3bdb5e6820e093bd2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032867b16cbae841e3bdb5e6820e093bd2 2023-04-03 05:54:43,737 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403289f3c0a2795488ca86a3ecfa541eab1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403289f3c0a2795488ca86a3ecfa541eab1 2023-04-03 05:54:43,738 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403329ba25658784a3c811de340b308cdbf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403329ba25658784a3c811de340b308cdbf 2023-04-03 05:54:43,739 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403399a58839bf3456a8496740d1e07e7eb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403399a58839bf3456a8496740d1e07e7eb 2023-04-03 05:54:43,740 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040349ebb56aa15541fd82ca72591c723ea5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040349ebb56aa15541fd82ca72591c723ea5 2023-04-03 05:54:43,740 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403548a8e53fea841a2aeb52ea40fa81a62 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403548a8e53fea841a2aeb52ea40fa81a62 2023-04-03 05:54:43,741 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036396070ed1434e0987fbd2f342cf55a3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036396070ed1434e0987fbd2f342cf55a3 2023-04-03 05:54:43,742 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036a4f464e00ed4a18a26c2638c090d0aa to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036a4f464e00ed4a18a26c2638c090d0aa 2023-04-03 05:54:43,743 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036f506ec39ef44f50a1e83002b81055f5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036f506ec39ef44f50a1e83002b81055f5 2023-04-03 05:54:43,744 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040383db8b9919c04261b470869c3865913e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040383db8b9919c04261b470869c3865913e 2023-04-03 05:54:43,745 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038621fd5519a94e4a99e15b30adac10a4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038621fd5519a94e4a99e15b30adac10a4 2023-04-03 05:54:43,746 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038fd8ce82e5e54013bf86b06b4a735c41 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038fd8ce82e5e54013bf86b06b4a735c41 2023-04-03 05:54:43,747 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403963f7380baec42d6bce3a1af72725558 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403963f7380baec42d6bce3a1af72725558 2023-04-03 05:54:43,748 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a1252c7d497246029ccb0e66e984d8a8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a1252c7d497246029ccb0e66e984d8a8 2023-04-03 05:54:43,749 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a2b0c4409970424bbf1e97f0c865ef09 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a2b0c4409970424bbf1e97f0c865ef09 2023-04-03 05:54:43,750 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a39f176d8e7b4d20a5bc32be0f3ae600 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a39f176d8e7b4d20a5bc32be0f3ae600 2023-04-03 05:54:43,751 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ad64bd4e443f455f8a6daa4413ce55cb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ad64bd4e443f455f8a6daa4413ce55cb 2023-04-03 05:54:43,752 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403be81890d15664f29a46a6e935dc91b47 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403be81890d15664f29a46a6e935dc91b47 2023-04-03 05:54:43,753 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c4bd8cbc4aa740efb004579574a0e1e2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c4bd8cbc4aa740efb004579574a0e1e2 2023-04-03 05:54:43,753 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d833ff8a45934773b501ae4f32a991a6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d833ff8a45934773b501ae4f32a991a6 2023-04-03 05:54:43,754 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403dd47a8e8e08d4e6784652140a2c013cd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403dd47a8e8e08d4e6784652140a2c013cd 2023-04-03 05:54:43,755 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 05:54:43,756 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:43,758 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 05:54:43,760 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 05:54:43,761 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:43,761 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 05:54:43,761 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680501283761"}]},"ts":"9223372036854775807"} 2023-04-03 05:54:43,762 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 05:54:43,762 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 574b09e98d501def4bb65b12ef4de07a, NAME => 'TestAcidGuarantees,,1680501248960.574b09e98d501def4bb65b12ef4de07a.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 05:54:43,762 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 05:54:43,762 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680501283762"}]},"ts":"9223372036854775807"} 2023-04-03 05:54:43,763 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 05:54:43,765 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:43,766 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 64 msec 2023-04-03 05:54:43,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-03 05:54:43,805 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-03 05:54:43,817 INFO [Listener at localhost/37275] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 230), OpenFileDescriptor=513 (was 557), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=336 (was 359), ProcessCount=169 (was 169), AvailableMemoryMB=3579 (was 3866) 2023-04-03 05:54:43,827 INFO [Listener at localhost/37275] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=513, MaxFileDescriptor=60000, SystemLoadAverage=336, ProcessCount=169, AvailableMemoryMB=3579 2023-04-03 05:54:43,829 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 05:54:43,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 05:54:43,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 05:54:43,831 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 05:54:43,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-03 05:54:43,831 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 05:54:43,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 05:54:43,832 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a 2023-04-03 05:54:43,833 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a empty. 2023-04-03 05:54:43,833 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a 2023-04-03 05:54:43,833 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 05:54:43,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 05:54:44,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 05:54:44,241 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 05:54:44,243 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 15471ccc940d59438142ba180c50365a, NAME => 'TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp 2023-04-03 05:54:44,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 05:54:44,648 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:54:44,648 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 15471ccc940d59438142ba180c50365a, disabling compactions & flushes 2023-04-03 05:54:44,648 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:44,648 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:44,648 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. after waiting 0 ms 2023-04-03 05:54:44,648 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:44,648 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:44,648 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:44,650 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 05:54:44,651 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501284651"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501284651"}]},"ts":"1680501284651"} 2023-04-03 05:54:44,652 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 05:54:44,653 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 05:54:44,653 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501284653"}]},"ts":"1680501284653"} 2023-04-03 05:54:44,654 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 05:54:44,656 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=15471ccc940d59438142ba180c50365a, ASSIGN}] 2023-04-03 05:54:44,658 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=15471ccc940d59438142ba180c50365a, ASSIGN 2023-04-03 05:54:44,658 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=15471ccc940d59438142ba180c50365a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37397,1680501214665; forceNewPlan=false, retain=false 2023-04-03 05:54:44,809 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=15471ccc940d59438142ba180c50365a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:44,810 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501284809"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501284809"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501284809"}]},"ts":"1680501284809"} 2023-04-03 05:54:44,811 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:54:44,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 05:54:44,965 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:44,965 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 15471ccc940d59438142ba180c50365a, NAME => 'TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.', STARTKEY => '', ENDKEY => ''} 2023-04-03 05:54:44,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:44,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:54:44,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:44,966 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:44,967 INFO [StoreOpener-15471ccc940d59438142ba180c50365a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:44,968 DEBUG [StoreOpener-15471ccc940d59438142ba180c50365a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A 2023-04-03 05:54:44,968 DEBUG [StoreOpener-15471ccc940d59438142ba180c50365a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A 2023-04-03 05:54:44,968 INFO [StoreOpener-15471ccc940d59438142ba180c50365a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:54:44,969 INFO [StoreOpener-15471ccc940d59438142ba180c50365a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 15471ccc940d59438142ba180c50365a columnFamilyName A 2023-04-03 05:54:44,969 INFO [StoreOpener-15471ccc940d59438142ba180c50365a-1] regionserver.HStore(310): Store=15471ccc940d59438142ba180c50365a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:54:44,969 INFO [StoreOpener-15471ccc940d59438142ba180c50365a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:44,970 DEBUG [StoreOpener-15471ccc940d59438142ba180c50365a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B 2023-04-03 05:54:44,970 DEBUG [StoreOpener-15471ccc940d59438142ba180c50365a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B 2023-04-03 05:54:44,970 INFO [StoreOpener-15471ccc940d59438142ba180c50365a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:54:44,971 INFO [StoreOpener-15471ccc940d59438142ba180c50365a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 15471ccc940d59438142ba180c50365a columnFamilyName B 2023-04-03 05:54:44,971 INFO [StoreOpener-15471ccc940d59438142ba180c50365a-1] regionserver.HStore(310): Store=15471ccc940d59438142ba180c50365a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:54:44,971 INFO [StoreOpener-15471ccc940d59438142ba180c50365a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:44,972 DEBUG [StoreOpener-15471ccc940d59438142ba180c50365a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C 2023-04-03 05:54:44,972 DEBUG [StoreOpener-15471ccc940d59438142ba180c50365a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C 2023-04-03 05:54:44,972 INFO [StoreOpener-15471ccc940d59438142ba180c50365a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:54:44,972 INFO [StoreOpener-15471ccc940d59438142ba180c50365a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 15471ccc940d59438142ba180c50365a columnFamilyName C 2023-04-03 05:54:44,973 INFO [StoreOpener-15471ccc940d59438142ba180c50365a-1] regionserver.HStore(310): Store=15471ccc940d59438142ba180c50365a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:54:44,973 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:44,973 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a 2023-04-03 05:54:44,974 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a 2023-04-03 05:54:44,975 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 05:54:44,976 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:44,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 05:54:44,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 15471ccc940d59438142ba180c50365a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=71686458, jitterRate=0.06821146607398987}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 05:54:44,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:44,978 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., pid=30, masterSystemTime=1680501284962 2023-04-03 05:54:44,979 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:44,979 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:44,980 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=15471ccc940d59438142ba180c50365a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:44,980 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501284980"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680501284980"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680501284980"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680501284980"}]},"ts":"1680501284980"} 2023-04-03 05:54:44,982 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-03 05:54:44,982 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 in 170 msec 2023-04-03 05:54:44,983 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-03 05:54:44,983 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=15471ccc940d59438142ba180c50365a, ASSIGN in 326 msec 2023-04-03 05:54:44,983 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 05:54:44,984 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501284983"}]},"ts":"1680501284983"} 2023-04-03 05:54:44,984 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 05:54:44,986 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 05:54:44,987 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-04-03 05:54:45,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-03 05:54:45,936 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-03 05:54:45,937 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x53921272 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:45,944 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d7cdc2b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:45,945 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:45,946 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41124, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:45,948 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 05:54:45,950 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:56386, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 05:54:45,951 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x13544701 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:45,955 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5096c69b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:45,955 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x10b79c2c to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:45,962 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@216aad74, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:45,962 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x5a5a81fa to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:45,965 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@76203061, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:45,966 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x3562c7bf to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:45,969 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b39d5b8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:45,970 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x6705d115 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:45,973 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44f9bd75, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:45,973 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x5e5b1c14 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:45,981 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@574000bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:45,981 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x4f29c1c4 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:45,985 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4994df89, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:45,985 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x681ace6c to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:45,989 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69bc154a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:45,989 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x2b4fa5d3 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:45,992 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15186778, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:45,993 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x45de5ad8 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:54:45,997 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@13e90e8d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:54:46,002 DEBUG [hconnection-0x2d6c6bde-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:46,002 DEBUG [hconnection-0x31355de6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:46,002 DEBUG [hconnection-0x3e29b42-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:46,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:46,003 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41168, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:46,004 DEBUG [hconnection-0x7c87340b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:46,005 DEBUG [hconnection-0x6d1102db-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:46,005 DEBUG [hconnection-0x4b79c4b5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:46,005 DEBUG [hconnection-0x5e1c22ea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:46,005 DEBUG [hconnection-0x44431a4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:46,005 DEBUG [hconnection-0x5223c760-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:46,005 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41152, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:46,005 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41136, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:46,005 DEBUG [hconnection-0x44247222-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:54:46,012 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41196, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:46,013 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41216, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:46,015 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41192, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:46,015 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41180, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:46,016 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41208, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:46,016 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41204, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:46,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:46,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:46,020 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:46,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:46,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:46,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:46,022 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:46,022 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:46,022 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:41230, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:54:46,029 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:46,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:46,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:46,029 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,029 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:46,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:46,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:46,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:46,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:46,031 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:46,031 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:46,031 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:46,031 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:46,031 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:46,031 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:46,031 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:46,032 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 05:54:46,034 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:46,034 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:46,034 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:46,034 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:46,034 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:46,034 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:46,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:46,035 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:46,050 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/8bdb3adacdb34a868d91c0462c9443ef 2023-04-03 05:54:46,067 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/f053bb06bb50476b8791898f4f9111b6 2023-04-03 05:54:46,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501286064, max timestamp=1680501286071Num uniques 859; 2023-04-03 05:54:46,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501286064, max timestamp=1680501286071Num uniques 859; 2023-04-03 05:54:46,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:46,074 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:46,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501286064, max timestamp=1680501286071Num uniques 859; 2023-04-03 05:54:46,076 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:46,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501346113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680501286108, max timestamp=1680501286114Num uniques 1018; 2023-04-03 05:54:46,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-03 05:54:46,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680501286108, max timestamp=1680501286114Num uniques 1018; 2023-04-03 05:54:46,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-03 05:54:46,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680501286108, max timestamp=1680501286114Num uniques 1018; 2023-04-03 05:54:46,118 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=12268, suffix heap size=133840,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=268 2023-04-03 05:54:46,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41230 deadline: 1680501346153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41204 deadline: 1680501346153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41216 deadline: 1680501346154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,156 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41168 deadline: 1680501346154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,220 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501346219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41230 deadline: 1680501346257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41204 deadline: 1680501346257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41216 deadline: 1680501346257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,258 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41168 deadline: 1680501346257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501346421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41204 deadline: 1680501346460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41230 deadline: 1680501346460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41168 deadline: 1680501346460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,462 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41216 deadline: 1680501346460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,494 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/1d59ba0f4a0c416eaaf3141b792edcd3 2023-04-03 05:54:46,501 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/8bdb3adacdb34a868d91c0462c9443ef as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/8bdb3adacdb34a868d91c0462c9443ef 2023-04-03 05:54:46,508 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/8bdb3adacdb34a868d91c0462c9443ef, entries=150, sequenceid=10, filesize=11.6 K 2023-04-03 05:54:46,509 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/f053bb06bb50476b8791898f4f9111b6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/f053bb06bb50476b8791898f4f9111b6 2023-04-03 05:54:46,514 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/f053bb06bb50476b8791898f4f9111b6, entries=150, sequenceid=10, filesize=11.6 K 2023-04-03 05:54:46,515 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/1d59ba0f4a0c416eaaf3141b792edcd3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/1d59ba0f4a0c416eaaf3141b792edcd3 2023-04-03 05:54:46,521 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/1d59ba0f4a0c416eaaf3141b792edcd3, entries=150, sequenceid=10, filesize=11.6 K 2023-04-03 05:54:46,522 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 15471ccc940d59438142ba180c50365a in 490ms, sequenceid=10, compaction requested=false 2023-04-03 05:54:46,522 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:46,522 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:46,522 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:46,523 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:46,523 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,523 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:46,523 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:46,524 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,524 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:46,525 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:46,525 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:46,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:46,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611d38e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:46,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:46,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:46,529 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:46,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:46,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:46,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:46,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:46,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:46,529 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,529 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:46,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:46,535 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:46,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:46,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:46,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:46,538 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:46,538 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:46,538 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:46,538 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:46,538 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:46,540 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:46,540 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:46,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:46,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:46,540 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:46,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:46,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:46,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:46,540 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:46,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:46,542 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:46,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:46,544 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,544 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:46,544 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:46,544 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,544 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:46,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:46,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:46,544 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:46,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:46,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:46,545 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:46,545 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:46,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:46,547 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:46,547 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:46,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:46,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:46,645 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:46,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:46,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:46,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:46,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:46,649 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:46,649 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:46,649 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:46,649 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:46,649 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:46,649 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:46,651 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,651 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:46,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:46,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:46,651 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,651 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:46,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:46,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:46,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:46,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:46,653 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:46,653 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:46,653 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:46,653 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:46,653 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:46,653 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:46,653 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:46,653 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=140.89 KB heapSize=394.69 KB 2023-04-03 05:54:46,653 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:46,653 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:46,653 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:46,653 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:46,653 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:46,653 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:46,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:46,725 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:46,741 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501346738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41230 deadline: 1680501346763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41204 deadline: 1680501346763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41168 deadline: 1680501346764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41216 deadline: 1680501346764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:46,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:46,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501346842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,045 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:47,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501347043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,062 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/e01131a8fd03449a875fac34946b4822 2023-04-03 05:54:47,073 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/5fdb028eccd142f1a86c866689cd70d4 2023-04-03 05:54:47,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:47,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41230 deadline: 1680501347266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:47,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41216 deadline: 1680501347268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:47,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41168 deadline: 1680501347269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:47,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41204 deadline: 1680501347269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,348 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:47,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501347348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,488 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/845fa9b92b5b4b549510ea98fe80d9e2 2023-04-03 05:54:47,493 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/e01131a8fd03449a875fac34946b4822 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e01131a8fd03449a875fac34946b4822 2023-04-03 05:54:47,498 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e01131a8fd03449a875fac34946b4822, entries=150, sequenceid=63, filesize=11.6 K 2023-04-03 05:54:47,502 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/5fdb028eccd142f1a86c866689cd70d4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5fdb028eccd142f1a86c866689cd70d4 2023-04-03 05:54:47,507 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5fdb028eccd142f1a86c866689cd70d4, entries=150, sequenceid=63, filesize=11.6 K 2023-04-03 05:54:47,508 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/845fa9b92b5b4b549510ea98fe80d9e2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/845fa9b92b5b4b549510ea98fe80d9e2 2023-04-03 05:54:47,512 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/845fa9b92b5b4b549510ea98fe80d9e2, entries=150, sequenceid=63, filesize=11.6 K 2023-04-03 05:54:47,512 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~394.64 KB/404112, currentSize=40.25 KB/41220 for 15471ccc940d59438142ba180c50365a in 859ms, sequenceid=63, compaction requested=false 2023-04-03 05:54:47,513 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:47,513 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:47,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:47,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:47,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:47,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:47,516 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28a633c1[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:47,519 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:47,520 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,521 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,521 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:47,521 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:47,521 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:47,521 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,521 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:47,523 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:47,523 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:47,523 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:47,523 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:47,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:47,527 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:47,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:47,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:47,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:47,527 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:47,529 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,529 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,529 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:47,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,531 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,533 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:47,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:47,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:47,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:47,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,539 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,539 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:47,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:47,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:47,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:47,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:47,539 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:47,539 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:47,639 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:47,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:47,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:47,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:47,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:47,644 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:47,644 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:47,644 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:47,644 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:47,645 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,645 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,646 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:47,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:47,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,647 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,647 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:47,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:47,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:47,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:47,649 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:47,649 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:47,649 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-04-03 05:54:47,650 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:47,650 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:47,650 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:47,650 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:47,650 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:47,650 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:47,659 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/1c4b5aa539f44ca8b039bb62ebe3f81c 2023-04-03 05:54:47,674 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/5bf6aefebc4b4e38a6addbf105641823 2023-04-03 05:54:47,700 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=72 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/932ff9894e3a4c85be1dcc6db4e6c9d7 2023-04-03 05:54:47,704 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/1c4b5aa539f44ca8b039bb62ebe3f81c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1c4b5aa539f44ca8b039bb62ebe3f81c 2023-04-03 05:54:47,709 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1c4b5aa539f44ca8b039bb62ebe3f81c, entries=100, sequenceid=72, filesize=9.3 K 2023-04-03 05:54:47,710 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/5bf6aefebc4b4e38a6addbf105641823 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5bf6aefebc4b4e38a6addbf105641823 2023-04-03 05:54:47,715 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5bf6aefebc4b4e38a6addbf105641823, entries=100, sequenceid=72, filesize=9.3 K 2023-04-03 05:54:47,716 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/932ff9894e3a4c85be1dcc6db4e6c9d7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/932ff9894e3a4c85be1dcc6db4e6c9d7 2023-04-03 05:54:47,720 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/932ff9894e3a4c85be1dcc6db4e6c9d7, entries=100, sequenceid=72, filesize=9.3 K 2023-04-03 05:54:47,721 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 15471ccc940d59438142ba180c50365a in 72ms, sequenceid=72, compaction requested=true 2023-04-03 05:54:47,721 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:47,721 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:47,721 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:47,723 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,723 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:47,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:47,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:47,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3823a784[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:47,725 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:47,725 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,726 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,727 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:47,727 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:47,727 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:47,727 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,727 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:47,729 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,729 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:47,729 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:47,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:47,729 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:47,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:47,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:47,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:47,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:47,732 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:47,732 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:47,732 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:47,732 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:47,732 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:47,734 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,734 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,734 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:47,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:47,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,735 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:47,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,737 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:47,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,739 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:47,739 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,739 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,739 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:47,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:47,740 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:47,739 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:47,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:47,739 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,740 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:47,740 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,740 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,740 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,740 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,740 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:47,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:47,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:47,840 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:47,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:47,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:47,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:47,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:47,844 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:47,844 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:47,844 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:47,844 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:47,845 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,845 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:47,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:47,846 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,847 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,847 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:47,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:47,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:47,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:47,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:47,848 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:47,852 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:47,852 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:47,852 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:47,852 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:47,855 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:47,855 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15e0cec3[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:47,857 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:47,857 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,859 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:47,859 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:47,859 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:47,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:47,861 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:47,861 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:47,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:47,861 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:47,862 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:47,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:47,864 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:47,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:47,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:47,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:47,864 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:47,866 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:47,866 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,866 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:47,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:47,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:47,867 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,870 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:47,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:47,871 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:54:47,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:47,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:47,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:47,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:47,871 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:47,871 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:47,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:47,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:47,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:47,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:47,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:47,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:47,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:47,877 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:47,877 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:47,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/b50d6e9c4c86428fa2bcb45311c8fe3c 2023-04-03 05:54:47,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501287928, max timestamp=1680501287938Num uniques 859; 2023-04-03 05:54:47,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501287928, max timestamp=1680501287938Num uniques 859; 2023-04-03 05:54:47,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:47,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:47,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501287928, max timestamp=1680501287938Num uniques 859; 2023-04-03 05:54:47,940 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:47,977 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:47,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:47,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:47,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:47,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:47,987 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:47,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:47,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:47,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:47,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,988 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:47,989 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:47,989 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,989 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:47,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:47,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:47,991 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:47,991 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:47,991 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:47,991 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,992 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:47,993 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,993 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:47,994 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1de5e252[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:47,994 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,995 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:47,995 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,995 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:47,998 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:48,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:48,000 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:48,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:48,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:48,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:48,001 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:48,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,004 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,005 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:48,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501348002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,007 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,008 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:48,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:48,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,008 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:48,008 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:48,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:48,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501348106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,108 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:48,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:48,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:48,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:48,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:48,112 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:48,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:48,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:48,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,114 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,114 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,114 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,114 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:48,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:48,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:48,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:48,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:48,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:48,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:48,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:48,115 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,115 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:48,116 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:48,116 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:48,116 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:48,119 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,119 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,119 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,119 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:48,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@444f214[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:48,121 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:48,121 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,122 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:48,122 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:48,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:48,123 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,123 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:48,126 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:48,126 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:48,126 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:48,126 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:48,126 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:48,127 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,127 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,127 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,128 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,129 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:48,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:48,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:48,132 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:48,132 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:48,132 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,232 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:48,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:48,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:48,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:48,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:48,236 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:48,236 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,236 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:48,236 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:48,236 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,236 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,237 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:48,238 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:48,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:48,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:48,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:48,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:48,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:48,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:48,239 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,239 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:48,240 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:48,240 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:48,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:48,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:48,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,241 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:48,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@217e6a9f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:48,243 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:48,243 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,244 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:48,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:48,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:48,244 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,244 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:48,246 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,246 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:48,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,246 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,247 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:48,249 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:48,249 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:48,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:48,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:48,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:48,249 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:48,250 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,250 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,250 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,251 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,252 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,257 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,257 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:48,257 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:48,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:48,257 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,257 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,258 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,258 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:48,258 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:48,258 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,258 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,258 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,258 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:48,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41204 deadline: 1680501348275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,277 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:48,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41230 deadline: 1680501348276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:48,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41168 deadline: 1680501348277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:48,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41216 deadline: 1680501348278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:48,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501348309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,312 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/212ccdba67cb44ba89e104271c914278 2023-04-03 05:54:48,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=83 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/ed5ac0ef6cd04a5ab1eba0a2c4193631 2023-04-03 05:54:48,331 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/b50d6e9c4c86428fa2bcb45311c8fe3c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b50d6e9c4c86428fa2bcb45311c8fe3c 2023-04-03 05:54:48,334 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b50d6e9c4c86428fa2bcb45311c8fe3c, entries=200, sequenceid=83, filesize=13.9 K 2023-04-03 05:54:48,335 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/212ccdba67cb44ba89e104271c914278 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/212ccdba67cb44ba89e104271c914278 2023-04-03 05:54:48,338 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/212ccdba67cb44ba89e104271c914278, entries=150, sequenceid=83, filesize=11.6 K 2023-04-03 05:54:48,339 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/ed5ac0ef6cd04a5ab1eba0a2c4193631 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/ed5ac0ef6cd04a5ab1eba0a2c4193631 2023-04-03 05:54:48,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/ed5ac0ef6cd04a5ab1eba0a2c4193631, entries=150, sequenceid=83, filesize=11.6 K 2023-04-03 05:54:48,343 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 15471ccc940d59438142ba180c50365a in 473ms, sequenceid=83, compaction requested=true 2023-04-03 05:54:48,343 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:48,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:48,343 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:48,344 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:48,344 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:48,344 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:48,345 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47540 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:48,345 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45200 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:48,345 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/A is initiating minor compaction (all files) 2023-04-03 05:54:48,345 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/B is initiating minor compaction (all files) 2023-04-03 05:54:48,345 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/A in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,345 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/B in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,345 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/8bdb3adacdb34a868d91c0462c9443ef, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e01131a8fd03449a875fac34946b4822, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1c4b5aa539f44ca8b039bb62ebe3f81c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b50d6e9c4c86428fa2bcb45311c8fe3c] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=46.4 K 2023-04-03 05:54:48,345 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/f053bb06bb50476b8791898f4f9111b6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5fdb028eccd142f1a86c866689cd70d4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5bf6aefebc4b4e38a6addbf105641823, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/212ccdba67cb44ba89e104271c914278] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=44.1 K 2023-04-03 05:54:48,345 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8bdb3adacdb34a868d91c0462c9443ef, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680501286028 2023-04-03 05:54:48,346 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting f053bb06bb50476b8791898f4f9111b6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680501286028 2023-04-03 05:54:48,346 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e01131a8fd03449a875fac34946b4822, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680501286148 2023-04-03 05:54:48,346 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 5fdb028eccd142f1a86c866689cd70d4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680501286148 2023-04-03 05:54:48,346 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 1c4b5aa539f44ca8b039bb62ebe3f81c, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1680501286732 2023-04-03 05:54:48,346 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting b50d6e9c4c86428fa2bcb45311c8fe3c, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680501287862 2023-04-03 05:54:48,347 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 5bf6aefebc4b4e38a6addbf105641823, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1680501286732 2023-04-03 05:54:48,347 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 212ccdba67cb44ba89e104271c914278, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680501287864 2023-04-03 05:54:48,353 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#A#compaction#248 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:48,354 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#B#compaction#249 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:48,358 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:48,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:48,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:48,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:48,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:48,372 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:48,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:48,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:48,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,378 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,378 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,379 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,379 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:48,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:48,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:48,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:48,380 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,380 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:48,380 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 05:54:48,380 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:48,381 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:48,381 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:48,381 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:48,381 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:48,381 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:48,390 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/32855fa86a3e4bbebd8a52479d8d3e29 2023-04-03 05:54:48,406 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/e276fde792dc49d1b28e3a02ae1d59e0 2023-04-03 05:54:48,434 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/2632cf60a9ec46078fb4ae509baa50c9 2023-04-03 05:54:48,439 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/32855fa86a3e4bbebd8a52479d8d3e29 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/32855fa86a3e4bbebd8a52479d8d3e29 2023-04-03 05:54:48,442 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/32855fa86a3e4bbebd8a52479d8d3e29, entries=150, sequenceid=120, filesize=11.6 K 2023-04-03 05:54:48,443 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/e276fde792dc49d1b28e3a02ae1d59e0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e276fde792dc49d1b28e3a02ae1d59e0 2023-04-03 05:54:48,447 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e276fde792dc49d1b28e3a02ae1d59e0, entries=150, sequenceid=120, filesize=11.6 K 2023-04-03 05:54:48,448 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/2632cf60a9ec46078fb4ae509baa50c9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/2632cf60a9ec46078fb4ae509baa50c9 2023-04-03 05:54:48,456 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/2632cf60a9ec46078fb4ae509baa50c9, entries=150, sequenceid=120, filesize=11.6 K 2023-04-03 05:54:48,457 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 15471ccc940d59438142ba180c50365a in 77ms, sequenceid=120, compaction requested=true 2023-04-03 05:54:48,457 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:48,457 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,457 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:48,457 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:48,457 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,457 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:48,457 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:48,459 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,459 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,459 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,459 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:48,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32fb2ea5[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:48,461 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:48,461 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,463 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:48,463 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:48,463 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,463 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,466 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:48,466 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:48,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:48,470 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:48,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:48,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:48,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:48,470 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:48,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,474 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,477 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,480 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:48,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:48,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,480 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:48,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,480 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:48,580 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:48,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:48,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:48,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:48,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:48,587 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:48,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:48,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:48,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,589 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,589 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,590 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:48,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:48,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:48,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:48,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:48,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:48,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:48,597 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:48,597 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:48,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:48,600 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,600 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,600 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,600 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:48,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:48,602 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:48,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@beadda[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:48,603 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,604 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:48,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:48,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:48,605 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:48,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:48,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:48,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:48,608 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:48,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:48,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:48,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:48,608 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:48,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,611 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:48,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:48,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:48,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,620 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:48,620 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:48,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:48,627 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:54:48,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:48,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:48,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:48,627 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:48,627 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:48,628 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:48,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501288684, max timestamp=1680501288694Num uniques 859; 2023-04-03 05:54:48,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:48,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501288684, max timestamp=1680501288694Num uniques 859; 2023-04-03 05:54:48,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:48,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501288684, max timestamp=1680501288694Num uniques 859; 2023-04-03 05:54:48,696 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:48,721 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:48,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:48,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:48,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:48,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:48,724 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:48,724 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,724 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:48,724 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:48,724 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,724 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,725 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,725 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:48,726 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:48,727 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:48,727 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:48,727 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:48,727 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,727 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:48,727 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:48,727 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:48,727 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,728 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:48,728 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:48,728 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:48,728 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:48,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:48,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:48,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23493b8b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:48,738 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:48,738 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,739 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,739 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:48,739 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:48,739 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:48,739 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,740 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:48,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:48,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,741 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:48,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:48,744 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:48,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:48,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:48,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:48,744 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:48,745 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,745 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,745 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,746 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,749 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,749 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,749 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,749 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,749 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,749 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:48,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:48,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:48,749 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,749 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,750 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:48,749 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,749 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,750 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:48,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:48,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501348770, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,776 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/1cd37b76702e4710a9325348ce1fdc5a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1cd37b76702e4710a9325348ce1fdc5a 2023-04-03 05:54:48,782 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/b089e71a59484f258b91827057955adc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b089e71a59484f258b91827057955adc 2023-04-03 05:54:48,784 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 15471ccc940d59438142ba180c50365a/A of 15471ccc940d59438142ba180c50365a into 1cd37b76702e4710a9325348ce1fdc5a(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:48,784 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:48,784 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/A, priority=12, startTime=1680501288343; duration=0sec 2023-04-03 05:54:48,785 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:48,785 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:54:48,786 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 15471ccc940d59438142ba180c50365a/B of 15471ccc940d59438142ba180c50365a into b089e71a59484f258b91827057955adc(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:48,786 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:48,786 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/B, priority=12, startTime=1680501288344; duration=0sec 2023-04-03 05:54:48,786 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:48,787 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:54:48,787 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/C is initiating minor compaction (all files) 2023-04-03 05:54:48,787 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/C in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,787 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/1d59ba0f4a0c416eaaf3141b792edcd3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/845fa9b92b5b4b549510ea98fe80d9e2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/932ff9894e3a4c85be1dcc6db4e6c9d7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/ed5ac0ef6cd04a5ab1eba0a2c4193631, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/2632cf60a9ec46078fb4ae509baa50c9] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=55.7 K 2023-04-03 05:54:48,787 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 1d59ba0f4a0c416eaaf3141b792edcd3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1680501286028 2023-04-03 05:54:48,788 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 845fa9b92b5b4b549510ea98fe80d9e2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680501286148 2023-04-03 05:54:48,788 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 932ff9894e3a4c85be1dcc6db4e6c9d7, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=72, earliestPutTs=1680501286732 2023-04-03 05:54:48,788 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting ed5ac0ef6cd04a5ab1eba0a2c4193631, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680501287864 2023-04-03 05:54:48,789 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 2632cf60a9ec46078fb4ae509baa50c9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680501287970 2023-04-03 05:54:48,796 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#C#compaction#254 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:48,850 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:48,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:48,852 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:48,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:48,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:48,853 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:48,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:48,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:48,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,858 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,858 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,858 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,858 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:48,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:48,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:48,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:48,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:48,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:48,859 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,860 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:48,860 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:48,860 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:48,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:48,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:48,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:48,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:48,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:48,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71b197ef[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:48,866 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:48,866 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,867 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:48,867 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:48,867 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:48,869 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,869 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:48,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:48,871 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:48,871 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:48,871 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:48,871 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:48,871 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:48,874 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,874 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,874 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,877 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,878 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:48,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501348877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,879 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,879 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,879 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,879 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:48,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:48,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:48,879 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,880 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:48,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,879 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:48,880 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:48,880 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:48,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:48,880 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,880 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,980 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:48,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:48,982 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:48,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:48,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:48,985 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:48,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:48,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:48,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,986 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,987 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:48,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:48,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:48,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:48,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:48,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:48,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:48,989 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,989 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:48,989 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:48,989 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:48,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:48,991 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:48,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:48,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ad4117a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:48,994 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:48,994 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,996 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:48,996 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,996 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:48,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:48,998 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,998 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:48,998 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:48,998 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:48,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:48,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:49,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:49,000 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:49,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:49,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:49,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:49,000 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:49,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:49,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:49,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:49,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,004 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,005 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:49,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:49,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:49,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:49,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,008 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:49,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:49,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:49,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,009 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:49,009 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:49,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:49,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:49,037 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/6f7ed9cff7b5465abe258a1a8dfa7fd7 2023-04-03 05:54:49,080 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:49,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501349079, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,109 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 05:54:49,109 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:49,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:49,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:49,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:49,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:49,112 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:49,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:49,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:49,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:49,113 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,113 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,115 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:49,115 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:49,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,116 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,116 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:49,116 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:49,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,116 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:49,117 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:49,117 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:49,117 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:49,117 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:49,117 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:49,117 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:49,117 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,118 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:49,118 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:49,119 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,119 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:49,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:49,119 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,120 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:49,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:49,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f71af87[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:49,121 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:49,121 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,122 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,122 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:49,123 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:49,124 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,124 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:49,124 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:49,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:49,124 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:49,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:49,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:49,128 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:49,128 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:49,128 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:49,128 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:49,128 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:49,131 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,131 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:49,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,131 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:49,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:49,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:49,132 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:49,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:49,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:49,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:49,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,137 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,137 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:49,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:49,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:49,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:49,137 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:49,137 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:49,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:49,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:49,204 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/18f7e3d55177470286ccd0f423cf7edf as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/18f7e3d55177470286ccd0f423cf7edf 2023-04-03 05:54:49,209 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 15471ccc940d59438142ba180c50365a/C of 15471ccc940d59438142ba180c50365a into 18f7e3d55177470286ccd0f423cf7edf(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:49,209 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:49,209 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/C, priority=11, startTime=1680501288344; duration=0sec 2023-04-03 05:54:49,209 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:49,237 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:49,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:49,239 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:49,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:49,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:49,240 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:49,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:49,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:49,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:49,241 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,241 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,242 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,242 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:49,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:49,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:49,242 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,243 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:49,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:49,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:49,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:49,244 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:49,244 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:49,244 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:49,244 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:49,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:49,246 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,246 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:49,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:49,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:49,246 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,246 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:49,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:49,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50e4481[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:49,248 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:49,248 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,250 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:49,250 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:49,250 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,250 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:49,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:49,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:49,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:49,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:49,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:49,254 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:49,254 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:49,254 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:49,254 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:49,254 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:49,256 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,256 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:49,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,256 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:49,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:49,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:49,257 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,258 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:49,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,260 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:49,260 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:49,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:49,260 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:49,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:49,260 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:49,261 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,261 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,261 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,261 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:49,261 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:49,261 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,261 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,261 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:49,361 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:49,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:49,363 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:49,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:49,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:49,364 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:49,364 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:49,364 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:49,364 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:49,365 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,365 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,367 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:49,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:49,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:49,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:49,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,368 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:49,368 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:49,368 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:49,368 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:49,368 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:49,368 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:49,368 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:49,368 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:49,368 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:49,368 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:49,369 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:49,369 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:49,369 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:49,370 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,370 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:49,370 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,370 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:49,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:49,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:49,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78fda520[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:49,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:49,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,373 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,373 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:49,373 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:49,373 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,373 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:49,373 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:49,374 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:49,374 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:49,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,374 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:49,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:49,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:49,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:49,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:49,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:49,376 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:49,377 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:49,377 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:49,377 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:49,377 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:49,380 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,380 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:49,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:49,380 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:49,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:49,381 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:49,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,382 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:49,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:49,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:49,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:49,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,384 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:49,384 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:49,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:49,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:49,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:49,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:49,384 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:49,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41208 deadline: 1680501349382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:49,385 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:49,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:49,385 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:49,447 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/82a401174f804377ad149730e94b2cab 2023-04-03 05:54:49,460 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=131 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/d248ca2f1f984b63b49100f1830ed68c 2023-04-03 05:54:49,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/6f7ed9cff7b5465abe258a1a8dfa7fd7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6f7ed9cff7b5465abe258a1a8dfa7fd7 2023-04-03 05:54:49,468 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6f7ed9cff7b5465abe258a1a8dfa7fd7, entries=200, sequenceid=131, filesize=14.0 K 2023-04-03 05:54:49,469 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/82a401174f804377ad149730e94b2cab as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/82a401174f804377ad149730e94b2cab 2023-04-03 05:54:49,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/82a401174f804377ad149730e94b2cab, entries=150, sequenceid=131, filesize=11.7 K 2023-04-03 05:54:49,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/d248ca2f1f984b63b49100f1830ed68c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d248ca2f1f984b63b49100f1830ed68c 2023-04-03 05:54:49,476 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d248ca2f1f984b63b49100f1830ed68c, entries=150, sequenceid=131, filesize=11.7 K 2023-04-03 05:54:49,477 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 15471ccc940d59438142ba180c50365a in 850ms, sequenceid=131, compaction requested=true 2023-04-03 05:54:49,477 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:49,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:49,477 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:49,477 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:49,477 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:49,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:49,478 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38236 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:49,478 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/A is initiating minor compaction (all files) 2023-04-03 05:54:49,478 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35896 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:49,478 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/A in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:49,479 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/B is initiating minor compaction (all files) 2023-04-03 05:54:49,479 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1cd37b76702e4710a9325348ce1fdc5a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/32855fa86a3e4bbebd8a52479d8d3e29, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6f7ed9cff7b5465abe258a1a8dfa7fd7] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=37.3 K 2023-04-03 05:54:49,479 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/B in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:49,479 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b089e71a59484f258b91827057955adc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e276fde792dc49d1b28e3a02ae1d59e0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/82a401174f804377ad149730e94b2cab] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=35.1 K 2023-04-03 05:54:49,479 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 1cd37b76702e4710a9325348ce1fdc5a, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680501287864 2023-04-03 05:54:49,479 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b089e71a59484f258b91827057955adc, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=83, earliestPutTs=1680501287864 2023-04-03 05:54:49,479 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 32855fa86a3e4bbebd8a52479d8d3e29, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680501287970 2023-04-03 05:54:49,480 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e276fde792dc49d1b28e3a02ae1d59e0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680501287970 2023-04-03 05:54:49,480 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 6f7ed9cff7b5465abe258a1a8dfa7fd7, keycount=200, bloomtype=ROW, size=14.0 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680501288614 2023-04-03 05:54:49,480 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 82a401174f804377ad149730e94b2cab, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680501288614 2023-04-03 05:54:49,493 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:49,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:49,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:49,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:49,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:49,498 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:49,498 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:49,498 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:49,498 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:49,498 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,498 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,499 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:49,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:49,500 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:49,501 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:49,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:49,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:49,501 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:49,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:49,524 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:49,524 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:49,524 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:49,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:49,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:49,535 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:49,535 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:49,535 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#B#compaction#257 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:49,535 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 05:54:49,536 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:49,536 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:49,537 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:49,537 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:49,537 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:49,538 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:49,541 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#A#compaction#258 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:49,619 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/2c8cb96d658447cda194342484d769ab 2023-04-03 05:54:49,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:49,890 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:49,914 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:49,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501349912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,013 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/f1540ce6a6834075bd65e101b5fc84cc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/f1540ce6a6834075bd65e101b5fc84cc 2023-04-03 05:54:50,016 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:50,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501350015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,017 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/A of 15471ccc940d59438142ba180c50365a into f1540ce6a6834075bd65e101b5fc84cc(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:50,017 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:50,018 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/A, priority=13, startTime=1680501289477; duration=0sec 2023-04-03 05:54:50,018 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:50,018 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-03 05:54:50,019 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 05:54:50,019 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 05:54:50,019 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. because compaction request was cancelled 2023-04-03 05:54:50,022 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/8011b9a1a58d427382b0029e9860598a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/8011b9a1a58d427382b0029e9860598a 2023-04-03 05:54:50,026 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/B of 15471ccc940d59438142ba180c50365a into 8011b9a1a58d427382b0029e9860598a(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:50,026 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:50,026 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/B, priority=13, startTime=1680501289477; duration=0sec 2023-04-03 05:54:50,026 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:50,038 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/0dca981c6d7c4620a2cd2187bc94ce07 2023-04-03 05:54:50,217 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:50,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501350217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:50,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41168 deadline: 1680501350282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,286 DEBUG [Thread-1870] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4130 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:54:50,289 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:50,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41230 deadline: 1680501350286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,290 DEBUG [Thread-1876] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4137 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:54:50,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:50,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41204 deadline: 1680501350293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:50,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41216 deadline: 1680501350293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,296 DEBUG [Thread-1872] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:54:50,297 DEBUG [Thread-1874] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:54:50,448 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=171 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/e086fd95baaa4578b6b51bad7d8785d2 2023-04-03 05:54:50,452 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/2c8cb96d658447cda194342484d769ab as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/2c8cb96d658447cda194342484d769ab 2023-04-03 05:54:50,456 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/2c8cb96d658447cda194342484d769ab, entries=150, sequenceid=171, filesize=11.8 K 2023-04-03 05:54:50,456 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/0dca981c6d7c4620a2cd2187bc94ce07 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/0dca981c6d7c4620a2cd2187bc94ce07 2023-04-03 05:54:50,461 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/0dca981c6d7c4620a2cd2187bc94ce07, entries=150, sequenceid=171, filesize=11.8 K 2023-04-03 05:54:50,462 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/e086fd95baaa4578b6b51bad7d8785d2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e086fd95baaa4578b6b51bad7d8785d2 2023-04-03 05:54:50,465 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e086fd95baaa4578b6b51bad7d8785d2, entries=150, sequenceid=171, filesize=11.8 K 2023-04-03 05:54:50,466 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 15471ccc940d59438142ba180c50365a in 930ms, sequenceid=171, compaction requested=true 2023-04-03 05:54:50,466 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:50,466 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:50,466 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:50,469 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:50,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:50,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:50,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:50,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42982484[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:50,471 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:50,471 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:50,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:50,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:50,472 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,472 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:50,473 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:50,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:50,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:50,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:50,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:50,476 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:50,476 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:50,476 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:50,476 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:50,476 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:50,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:50,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:50,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:50,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,480 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:50,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:50,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:50,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:50,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:50,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:50,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,484 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:50,484 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:50,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:50,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:50,521 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 05:54:50,521 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:50,521 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:50,522 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:50,522 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:50,522 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:50,522 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:50,531 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/4652cf9b25234ee3abd18bbf61cc67c3 2023-04-03 05:54:50,545 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/6f7423637b454e2a829720f65bbce824 2023-04-03 05:54:50,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501290564, max timestamp=1680501290572Num uniques 859; 2023-04-03 05:54:50,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501290564, max timestamp=1680501290572Num uniques 859; 2023-04-03 05:54:50,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501290564, max timestamp=1680501290572Num uniques 859; 2023-04-03 05:54:50,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:50,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:50,574 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:54:50,584 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:50,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:50,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:50,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:50,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:50,587 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:50,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:50,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:50,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:50,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,589 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:50,589 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:50,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:50,590 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,590 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:50,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:50,591 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:50,591 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:50,591 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:50,591 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:50,591 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:50,591 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:50,591 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:50,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:50,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:50,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:50,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20f7a6c7[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:50,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:50,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,597 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:50,597 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,597 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:50,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:50,601 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:50,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:50,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:50,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:50,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:50,603 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:50,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:50,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:50,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:50,604 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:50,606 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,606 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:50,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,606 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:50,606 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:50,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,607 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:50,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:50,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:50,610 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:50,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:50,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:50,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:50,610 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:50,611 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:50,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:50,621 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:50,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501350619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,711 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:50,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:50,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:50,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:50,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:50,714 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:50,714 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:50,714 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:50,714 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:50,715 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,715 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:50,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:50,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:50,717 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,717 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,717 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,717 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:50,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,718 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:50,718 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:50,718 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:50,718 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:50,718 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:50,718 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:50,718 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:50,718 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:50,719 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:50,720 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,720 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:50,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:50,721 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,721 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:50,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57aad414[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:50,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:50,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501350722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,724 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:50,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:50,725 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,725 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:50,726 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:50,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:50,726 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:50,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:50,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:50,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:50,729 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:50,729 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:50,729 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:50,729 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:50,729 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:50,731 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,731 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,731 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:50,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:50,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:50,732 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,733 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,736 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:50,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:50,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:50,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:50,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:50,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:50,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:50,736 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:50,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:50,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,736 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:50,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:50,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:50,836 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:50,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:50,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:50,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:50,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:50,848 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:50,848 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:50,848 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:50,848 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:50,849 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,849 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,850 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,850 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:50,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:50,850 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,850 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:50,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:50,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:50,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:50,851 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:50,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:50,852 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:50,852 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:50,852 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:50,852 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:50,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:50,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:50,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:50,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:50,853 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,853 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,854 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:50,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:50,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:50,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@426bbf64[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:50,855 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:50,855 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,856 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:50,857 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:50,857 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:50,857 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,857 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:50,859 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:50,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:50,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:50,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:50,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:50,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:50,861 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:50,861 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:50,861 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:50,861 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:50,861 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:50,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:50,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:50,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:50,863 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:50,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:50,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:50,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,866 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,866 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,867 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,867 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,867 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,867 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:50,867 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,867 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:50,867 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:50,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:50,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:50,867 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:50,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:50,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:50,867 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:50,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:50,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:50,867 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:50,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:50,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501350923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,955 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/8db7387285de432682be11570b3e929a 2023-04-03 05:54:50,959 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/4652cf9b25234ee3abd18bbf61cc67c3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/4652cf9b25234ee3abd18bbf61cc67c3 2023-04-03 05:54:50,963 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/4652cf9b25234ee3abd18bbf61cc67c3, entries=200, sequenceid=184, filesize=14.1 K 2023-04-03 05:54:50,964 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/6f7423637b454e2a829720f65bbce824 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6f7423637b454e2a829720f65bbce824 2023-04-03 05:54:50,967 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6f7423637b454e2a829720f65bbce824, entries=150, sequenceid=184, filesize=11.8 K 2023-04-03 05:54:50,967 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:50,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:50,968 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/8db7387285de432682be11570b3e929a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8db7387285de432682be11570b3e929a 2023-04-03 05:54:50,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:50,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:50,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:50,971 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:50,971 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:50,971 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:50,971 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:50,971 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,972 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,972 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8db7387285de432682be11570b3e929a, entries=150, sequenceid=184, filesize=11.8 K 2023-04-03 05:54:50,973 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 15471ccc940d59438142ba180c50365a in 452ms, sequenceid=184, compaction requested=true 2023-04-03 05:54:50,973 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:50,973 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:50,973 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:50,973 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:50,973 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:50,973 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:50,975 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:50,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:50,975 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,975 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:50,976 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:50,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:50,976 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:50,976 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38688 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:50,977 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/A is initiating minor compaction (all files) 2023-04-03 05:54:50,977 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/A in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:50,977 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/f1540ce6a6834075bd65e101b5fc84cc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/2c8cb96d658447cda194342484d769ab, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/4652cf9b25234ee3abd18bbf61cc67c3] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=37.8 K 2023-04-03 05:54:50,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:50,977 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:50,977 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36298 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:50,977 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:50,977 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:50,977 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting f1540ce6a6834075bd65e101b5fc84cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680501288614 2023-04-03 05:54:50,977 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-03 05:54:50,977 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:50,977 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/B is initiating minor compaction (all files) 2023-04-03 05:54:50,978 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/B in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:50,978 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:50,978 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/8011b9a1a58d427382b0029e9860598a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/0dca981c6d7c4620a2cd2187bc94ce07, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6f7423637b454e2a829720f65bbce824] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=35.4 K 2023-04-03 05:54:50,978 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:50,978 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:50,978 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:50,978 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:50,978 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:54:50,978 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 2c8cb96d658447cda194342484d769ab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1680501288716 2023-04-03 05:54:50,979 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 8011b9a1a58d427382b0029e9860598a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680501288614 2023-04-03 05:54:50,979 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 4652cf9b25234ee3abd18bbf61cc67c3, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680501289894 2023-04-03 05:54:50,979 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0dca981c6d7c4620a2cd2187bc94ce07, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1680501288716 2023-04-03 05:54:50,980 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6f7423637b454e2a829720f65bbce824, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680501289894 2023-04-03 05:54:50,989 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#A#compaction#266 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:50,997 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#B#compaction#267 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:51,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:51,228 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:51,268 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:51,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501351266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:51,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501351369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,402 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/001718739bc047fab573e19a1e395b1a 2023-04-03 05:54:51,412 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/29be80cc10ba4dc3a3ec027b67efcd43 2023-04-03 05:54:51,427 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/c69ffd267ff24fb2be37cf88ed6aa567 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/c69ffd267ff24fb2be37cf88ed6aa567 2023-04-03 05:54:51,430 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/19e65f306d7b442b9f4b57514471047d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/19e65f306d7b442b9f4b57514471047d 2023-04-03 05:54:51,433 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/B of 15471ccc940d59438142ba180c50365a into c69ffd267ff24fb2be37cf88ed6aa567(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:51,433 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:51,433 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/B, priority=13, startTime=1680501290973; duration=0sec 2023-04-03 05:54:51,433 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:51,433 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:54:51,443 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48116 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:54:51,443 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/C is initiating minor compaction (all files) 2023-04-03 05:54:51,443 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/C in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:51,443 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/18f7e3d55177470286ccd0f423cf7edf, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d248ca2f1f984b63b49100f1830ed68c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e086fd95baaa4578b6b51bad7d8785d2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8db7387285de432682be11570b3e929a] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=47.0 K 2023-04-03 05:54:51,444 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 18f7e3d55177470286ccd0f423cf7edf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680501287970 2023-04-03 05:54:51,444 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/A of 15471ccc940d59438142ba180c50365a into 19e65f306d7b442b9f4b57514471047d(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:51,444 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:51,444 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/A, priority=13, startTime=1680501290973; duration=0sec 2023-04-03 05:54:51,444 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting d248ca2f1f984b63b49100f1830ed68c, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=131, earliestPutTs=1680501288614 2023-04-03 05:54:51,444 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:51,445 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e086fd95baaa4578b6b51bad7d8785d2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=171, earliestPutTs=1680501288716 2023-04-03 05:54:51,445 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 8db7387285de432682be11570b3e929a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680501289894 2023-04-03 05:54:51,451 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#C#compaction#270 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:51,573 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:51,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501351572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,822 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/24ea684fecb64209b54e65807e827e8c 2023-04-03 05:54:51,826 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/001718739bc047fab573e19a1e395b1a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/001718739bc047fab573e19a1e395b1a 2023-04-03 05:54:51,830 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/001718739bc047fab573e19a1e395b1a, entries=150, sequenceid=220, filesize=11.8 K 2023-04-03 05:54:51,831 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/29be80cc10ba4dc3a3ec027b67efcd43 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/29be80cc10ba4dc3a3ec027b67efcd43 2023-04-03 05:54:51,834 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/29be80cc10ba4dc3a3ec027b67efcd43, entries=150, sequenceid=220, filesize=11.8 K 2023-04-03 05:54:51,835 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/24ea684fecb64209b54e65807e827e8c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/24ea684fecb64209b54e65807e827e8c 2023-04-03 05:54:51,839 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/24ea684fecb64209b54e65807e827e8c, entries=150, sequenceid=220, filesize=11.8 K 2023-04-03 05:54:51,839 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 15471ccc940d59438142ba180c50365a in 862ms, sequenceid=220, compaction requested=false 2023-04-03 05:54:51,839 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:51,839 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:51,839 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:51,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,842 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:51,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41d41fa9[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:51,844 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:51,844 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,846 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:51,846 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:51,846 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:51,846 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,846 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:51,847 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,847 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:51,848 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:51,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:51,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:51,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:51,850 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:51,850 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:51,850 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:51,850 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:51,850 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:51,852 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,852 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:51,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,852 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:51,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:51,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:51,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:51,854 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,860 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/8a09213cb3c945b58f9a5c528a318320 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8a09213cb3c945b58f9a5c528a318320 2023-04-03 05:54:51,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:51,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:51,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:51,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:51,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:51,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:51,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:51,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:51,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:51,863 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:51,864 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:51,864 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:51,867 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 15471ccc940d59438142ba180c50365a/C of 15471ccc940d59438142ba180c50365a into 8a09213cb3c945b58f9a5c528a318320(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:51,867 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:51,867 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/C, priority=12, startTime=1680501290973; duration=0sec 2023-04-03 05:54:51,867 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:51,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:51,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:51,877 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 05:54:51,877 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:51,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:51,877 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:51,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:51,877 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:51,877 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:51,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:51,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501351921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,964 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:51,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:51,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:51,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:51,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:51,967 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:51,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:51,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:51,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:51,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:51,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,968 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:51,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:51,969 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,969 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:51,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:51,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:51,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:51,970 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:51,970 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:51,970 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:51,970 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:51,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:51,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:51,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:51,972 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:51,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,972 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:51,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:51,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c0858ca[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:51,974 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:51,974 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,975 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,975 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:51,975 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:51,975 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,975 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:51,975 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:51,977 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,977 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:51,977 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:51,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:51,977 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:51,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:51,980 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:51,980 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:51,980 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:51,980 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:51,980 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:51,980 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:51,981 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,982 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,982 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:51,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:51,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:51,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,983 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:51,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,984 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:51,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:51,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:51,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:51,986 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:51,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:51,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:51,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:51,986 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:51,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:51,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:51,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:51,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:51,986 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:51,986 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:51,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:51,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:52,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:52,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501352024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,087 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:52,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:52,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:52,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:52,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:52,090 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:52,090 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:52,090 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:52,090 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:52,090 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,090 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,091 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,091 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:52,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:52,092 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,092 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,092 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,093 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:52,093 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:52,093 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,093 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:52,094 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:52,094 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,094 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:52,094 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:52,094 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,094 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:52,094 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:52,096 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,096 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,097 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d833d2b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:52,098 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:52,098 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,099 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:52,099 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:52,100 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:52,100 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,100 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:52,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:52,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:52,101 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:52,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:52,104 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:52,104 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:52,104 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:52,104 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:52,104 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:52,107 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,108 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:52,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,108 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,113 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,116 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,117 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,117 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,117 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,117 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,117 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:52,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:52,117 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:52,117 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:52,117 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:52,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:52,117 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:52,117 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:52,117 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:52,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:52,217 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:52,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:52,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:52,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:52,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:52,221 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:52,221 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:52,221 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:52,221 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:52,221 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,221 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:52,223 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:52,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:52,223 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,223 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:52,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:52,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:52,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:52,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:52,225 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,225 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:52,225 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:52,225 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:52,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:52,227 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:52,227 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501352225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,227 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,227 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,227 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:52,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@428c3960[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:52,229 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:52,229 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,230 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:52,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:52,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:52,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:52,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:52,238 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:52,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:52,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:52,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:52,238 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:52,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:52,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,242 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,243 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,247 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:52,247 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,248 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:52,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:52,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:52,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:52,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:52,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:52,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,248 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:52,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,248 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:52,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:52,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:52,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,283 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/54b3980dafc945b19bf01d1d5cfd4f19 2023-04-03 05:54:52,294 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/b779dfe2ee2a41c08139751cfe46dfb3 2023-04-03 05:54:52,348 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:52,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:52,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:52,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:52,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:52,352 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:52,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:52,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:52,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:52,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:52,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:52,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,355 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,355 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:52,355 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:52,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:52,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:52,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:52,357 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,357 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:52,357 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:52,357 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:52,358 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,358 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:52,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23c45fcf[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:52,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:52,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,362 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:52,362 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:52,362 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:52,362 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,362 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:52,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:52,363 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:52,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:52,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:52,366 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:52,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:52,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:52,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:52,367 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:52,369 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,369 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,369 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:52,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:52,370 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,371 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,373 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:52,373 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:52,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:52,373 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:52,373 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:52,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:52,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:52,374 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:52,374 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:52,374 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:52,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,374 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:52,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:52,474 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:52,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:52,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:52,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:52,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:52,477 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:52,477 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:52,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:52,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:52,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:52,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:52,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:52,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:52,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:52,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:52,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:52,481 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,481 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:52,481 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:52,481 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:52,483 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,483 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,483 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:52,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d948d75[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:52,484 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:52,484 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,486 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:52,487 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,487 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:52,488 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:52,490 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:52,490 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:52,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:52,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:52,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:52,490 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:52,491 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,491 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,491 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:52,492 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:52,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,496 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:52,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:52,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:52,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:52,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:52,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:52,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,496 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:52,496 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:52,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:52,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:52,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501352528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,596 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:52,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:52,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:52,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:52,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:52,599 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:52,599 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:52,600 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:52,600 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:52,600 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,600 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,602 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:52,602 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:52,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:52,602 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,603 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,603 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:52,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:52,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:52,604 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,604 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:52,604 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:52,604 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:52,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:52,606 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,606 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:52,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:52,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79ba8589[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:52,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:52,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,609 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,609 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:52,609 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:52,609 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:52,610 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,610 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:52,611 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:52,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:52,625 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:52,625 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:52,625 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:52,625 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:52,625 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:52,627 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,627 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:52,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,628 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:52,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:52,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,629 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:52,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,635 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,635 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,635 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,636 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:52,636 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,636 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:52,636 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:52,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:52,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:52,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:52,636 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,636 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:52,636 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,636 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:52,637 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:52,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:52,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:52,706 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=235 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/a3943087fd4a4c7493489e76ce9a33b5 2023-04-03 05:54:52,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/54b3980dafc945b19bf01d1d5cfd4f19 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/54b3980dafc945b19bf01d1d5cfd4f19 2023-04-03 05:54:52,715 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/54b3980dafc945b19bf01d1d5cfd4f19, entries=200, sequenceid=235, filesize=14.1 K 2023-04-03 05:54:52,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/b779dfe2ee2a41c08139751cfe46dfb3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b779dfe2ee2a41c08139751cfe46dfb3 2023-04-03 05:54:52,719 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b779dfe2ee2a41c08139751cfe46dfb3, entries=150, sequenceid=235, filesize=11.8 K 2023-04-03 05:54:52,720 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/a3943087fd4a4c7493489e76ce9a33b5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/a3943087fd4a4c7493489e76ce9a33b5 2023-04-03 05:54:52,723 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/a3943087fd4a4c7493489e76ce9a33b5, entries=150, sequenceid=235, filesize=11.8 K 2023-04-03 05:54:52,724 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 15471ccc940d59438142ba180c50365a in 847ms, sequenceid=235, compaction requested=true 2023-04-03 05:54:52,724 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:52,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:52,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:52,724 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:52,724 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:52,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:52,725 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38840 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:52,725 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/A is initiating minor compaction (all files) 2023-04-03 05:54:52,725 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36450 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:52,726 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/B is initiating minor compaction (all files) 2023-04-03 05:54:52,726 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/A in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,726 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/B in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,726 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/19e65f306d7b442b9f4b57514471047d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/001718739bc047fab573e19a1e395b1a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/54b3980dafc945b19bf01d1d5cfd4f19] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=37.9 K 2023-04-03 05:54:52,726 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/c69ffd267ff24fb2be37cf88ed6aa567, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/29be80cc10ba4dc3a3ec027b67efcd43, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b779dfe2ee2a41c08139751cfe46dfb3] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=35.6 K 2023-04-03 05:54:52,726 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 19e65f306d7b442b9f4b57514471047d, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680501289894 2023-04-03 05:54:52,726 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting c69ffd267ff24fb2be37cf88ed6aa567, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680501289894 2023-04-03 05:54:52,727 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 29be80cc10ba4dc3a3ec027b67efcd43, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680501290607 2023-04-03 05:54:52,727 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 001718739bc047fab573e19a1e395b1a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680501290607 2023-04-03 05:54:52,727 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b779dfe2ee2a41c08139751cfe46dfb3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680501291262 2023-04-03 05:54:52,727 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 54b3980dafc945b19bf01d1d5cfd4f19, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680501291256 2023-04-03 05:54:52,735 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#B#compaction#274 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:52,735 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#A#compaction#275 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:52,737 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:52,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:52,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:52,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:52,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:52,741 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:52,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:52,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:52,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:52,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,745 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/85a4e289b3b74ca98e301f264842348e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/85a4e289b3b74ca98e301f264842348e 2023-04-03 05:54:52,746 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:52,746 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:52,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:52,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:52,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:52,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,747 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:52,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:52,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:52,748 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:52,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:52,748 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:52,748 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:52,748 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:52,748 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:52,748 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:52,748 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,748 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:52,749 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 05:54:52,749 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:52,749 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:52,749 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:52,749 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:52,749 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:52,749 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:52,750 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/B of 15471ccc940d59438142ba180c50365a into 85a4e289b3b74ca98e301f264842348e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:52,750 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:52,750 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/B, priority=13, startTime=1680501292724; duration=0sec 2023-04-03 05:54:52,750 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:52,750 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:52,751 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36416 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:52,751 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/C is initiating minor compaction (all files) 2023-04-03 05:54:52,751 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/C in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:52,751 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8a09213cb3c945b58f9a5c528a318320, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/24ea684fecb64209b54e65807e827e8c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/a3943087fd4a4c7493489e76ce9a33b5] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=35.6 K 2023-04-03 05:54:52,752 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 8a09213cb3c945b58f9a5c528a318320, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680501289894 2023-04-03 05:54:52,752 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 24ea684fecb64209b54e65807e827e8c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680501290607 2023-04-03 05:54:52,752 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a3943087fd4a4c7493489e76ce9a33b5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680501291262 2023-04-03 05:54:52,756 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/e2546f94bf34435c95081a265a237511 2023-04-03 05:54:52,758 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#C#compaction#277 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:53,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:53,035 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:53,058 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:53,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501353056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,146 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/7a655be4011f42279e740dd95c1ea3f1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7a655be4011f42279e740dd95c1ea3f1 2023-04-03 05:54:53,151 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/A of 15471ccc940d59438142ba180c50365a into 7a655be4011f42279e740dd95c1ea3f1(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:53,151 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:53,151 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/A, priority=13, startTime=1680501292724; duration=0sec 2023-04-03 05:54:53,151 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:53,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:53,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501353159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,169 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/4b13fc2d62cc478ea8e57045d372935d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/4b13fc2d62cc478ea8e57045d372935d 2023-04-03 05:54:53,173 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/C of 15471ccc940d59438142ba180c50365a into 4b13fc2d62cc478ea8e57045d372935d(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:53,173 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:53,173 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/C, priority=13, startTime=1680501292724; duration=0sec 2023-04-03 05:54:53,173 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:53,178 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/e2f66c62d2cc4a689710076a6962ef9f 2023-04-03 05:54:53,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:53,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501353364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,588 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=255 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/bf541ce05a08409f95b8513da2ef6e95 2023-04-03 05:54:53,594 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/e2546f94bf34435c95081a265a237511 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e2546f94bf34435c95081a265a237511 2023-04-03 05:54:53,600 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e2546f94bf34435c95081a265a237511, entries=150, sequenceid=255, filesize=11.8 K 2023-04-03 05:54:53,601 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/e2f66c62d2cc4a689710076a6962ef9f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e2f66c62d2cc4a689710076a6962ef9f 2023-04-03 05:54:53,626 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e2f66c62d2cc4a689710076a6962ef9f, entries=150, sequenceid=255, filesize=11.8 K 2023-04-03 05:54:53,628 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/bf541ce05a08409f95b8513da2ef6e95 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/bf541ce05a08409f95b8513da2ef6e95 2023-04-03 05:54:53,634 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/bf541ce05a08409f95b8513da2ef6e95, entries=150, sequenceid=255, filesize=11.8 K 2023-04-03 05:54:53,634 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 15471ccc940d59438142ba180c50365a in 886ms, sequenceid=255, compaction requested=false 2023-04-03 05:54:53,634 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:53,635 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:53,635 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:53,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,640 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:53,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:53,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:53,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:53,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a5ef4e7[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:53,647 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:53,647 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,648 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,648 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:53,648 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:53,648 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:53,648 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,648 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:53,650 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,650 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:53,650 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:53,650 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:53,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:53,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:53,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:53,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:53,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:53,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:53,653 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:53,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:53,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:53,654 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:53,654 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:53,657 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,658 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:53,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,658 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:53,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:53,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:53,659 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,661 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:53,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:53,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:53,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:53,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,665 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:53,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:53,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:53,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:53,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:53,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:53,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:53,666 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:53,666 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:53,674 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 05:54:53,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:53,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:53,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:53,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:53,674 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:53,674 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:53,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:53,714 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/40c96ae5a10f4c5f8af4c3d5fffdcc94 2023-04-03 05:54:53,748 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/42691ec04d354585b5e95287008bbc27 2023-04-03 05:54:53,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:53,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501353750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,766 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:53,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:53,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:53,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:53,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:53,788 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:53,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:53,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:53,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:53,789 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,789 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:53,789 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=271 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/5028cfeb1abe4a28ba7850691d91c91e 2023-04-03 05:54:53,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:53,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:53,790 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:53,791 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,791 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:53,792 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:53,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:53,795 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/40c96ae5a10f4c5f8af4c3d5fffdcc94 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/40c96ae5a10f4c5f8af4c3d5fffdcc94 2023-04-03 05:54:53,800 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:53,800 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:53,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:53,801 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:53,801 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:53,801 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:53,801 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/40c96ae5a10f4c5f8af4c3d5fffdcc94, entries=200, sequenceid=271, filesize=14.3 K 2023-04-03 05:54:53,802 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:53,803 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:53,803 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:53,803 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,803 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/42691ec04d354585b5e95287008bbc27 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/42691ec04d354585b5e95287008bbc27 2023-04-03 05:54:53,804 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:53,804 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:53,806 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:53,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:53,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:53,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:53,808 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5be5f244[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:53,808 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:53,808 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,809 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:53,810 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:53,810 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,810 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:53,811 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:53,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:53,812 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:53,812 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:53,812 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:53,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:53,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:53,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:53,814 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:53,814 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:53,814 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:53,814 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:53,814 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:53,815 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/42691ec04d354585b5e95287008bbc27, entries=150, sequenceid=271, filesize=11.9 K 2023-04-03 05:54:53,816 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/5028cfeb1abe4a28ba7850691d91c91e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/5028cfeb1abe4a28ba7850691d91c91e 2023-04-03 05:54:53,817 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,817 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,817 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:53,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:53,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:53,818 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,821 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/5028cfeb1abe4a28ba7850691d91c91e, entries=150, sequenceid=271, filesize=11.9 K 2023-04-03 05:54:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:53,822 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 15471ccc940d59438142ba180c50365a in 148ms, sequenceid=271, compaction requested=true 2023-04-03 05:54:53,822 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:53,822 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:53,822 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:53,822 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:53,822 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:53,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,822 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:53,823 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39142 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:53,823 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/A is initiating minor compaction (all files) 2023-04-03 05:54:53,823 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36702 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:53,823 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/A in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:53,823 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/B is initiating minor compaction (all files) 2023-04-03 05:54:53,823 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7a655be4011f42279e740dd95c1ea3f1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e2546f94bf34435c95081a265a237511, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/40c96ae5a10f4c5f8af4c3d5fffdcc94] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=38.2 K 2023-04-03 05:54:53,823 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/B in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:53,823 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/85a4e289b3b74ca98e301f264842348e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e2f66c62d2cc4a689710076a6962ef9f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/42691ec04d354585b5e95287008bbc27] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=35.8 K 2023-04-03 05:54:53,823 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:53,823 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:53,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:53,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:53,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:53,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,824 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:53,824 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 7a655be4011f42279e740dd95c1ea3f1, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680501291262 2023-04-03 05:54:53,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:53,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:53,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:53,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,824 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:53,824 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:53,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:53,824 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 85a4e289b3b74ca98e301f264842348e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680501291262 2023-04-03 05:54:53,824 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e2546f94bf34435c95081a265a237511, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680501291909 2023-04-03 05:54:53,825 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e2f66c62d2cc4a689710076a6962ef9f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680501291909 2023-04-03 05:54:53,825 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 40c96ae5a10f4c5f8af4c3d5fffdcc94, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680501293049 2023-04-03 05:54:53,825 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 42691ec04d354585b5e95287008bbc27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680501293049 2023-04-03 05:54:53,831 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#B#compaction#283 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:53,831 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#A#compaction#284 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:53,847 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/e60664a5080c47109d7d6d1ae6377af0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e60664a5080c47109d7d6d1ae6377af0 2023-04-03 05:54:53,852 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/B of 15471ccc940d59438142ba180c50365a into e60664a5080c47109d7d6d1ae6377af0(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:53,852 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:53,852 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/B, priority=13, startTime=1680501293822; duration=0sec 2023-04-03 05:54:53,852 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:53,852 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:53,853 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36668 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:53,853 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/C is initiating minor compaction (all files) 2023-04-03 05:54:53,853 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/C in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:53,853 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/4b13fc2d62cc478ea8e57045d372935d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/bf541ce05a08409f95b8513da2ef6e95, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/5028cfeb1abe4a28ba7850691d91c91e] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=35.8 K 2023-04-03 05:54:53,854 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 4b13fc2d62cc478ea8e57045d372935d, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=235, earliestPutTs=1680501291262 2023-04-03 05:54:53,854 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting bf541ce05a08409f95b8513da2ef6e95, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=255, earliestPutTs=1680501291909 2023-04-03 05:54:53,855 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 5028cfeb1abe4a28ba7850691d91c91e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680501293049 2023-04-03 05:54:53,856 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 05:54:53,856 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:53,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:53,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:53,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:53,857 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:53,857 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:53,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:53,865 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#C#compaction#286 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:53,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/1d0cd8cabc37461e99af945f6c16fd63 2023-04-03 05:54:53,887 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/7f51f28963974c7fa2df5a4d4bcc8833 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/7f51f28963974c7fa2df5a4d4bcc8833 2023-04-03 05:54:53,888 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:53,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501353886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,891 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/C of 15471ccc940d59438142ba180c50365a into 7f51f28963974c7fa2df5a4d4bcc8833(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:53,891 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:53,891 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/C, priority=13, startTime=1680501293822; duration=0sec 2023-04-03 05:54:53,891 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:53,913 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/6f3c249af15d4d199cb5b25df2bff4c6 2023-04-03 05:54:53,923 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=292 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/c6a92b601d3f4904ae46ae1470c56c8d 2023-04-03 05:54:53,924 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:53,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:53,927 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/1d0cd8cabc37461e99af945f6c16fd63 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1d0cd8cabc37461e99af945f6c16fd63 2023-04-03 05:54:53,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:53,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:53,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:53,928 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:53,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:53,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:53,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:53,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:53,931 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1d0cd8cabc37461e99af945f6c16fd63, entries=150, sequenceid=292, filesize=11.9 K 2023-04-03 05:54:53,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/6f3c249af15d4d199cb5b25df2bff4c6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6f3c249af15d4d199cb5b25df2bff4c6 2023-04-03 05:54:53,936 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6f3c249af15d4d199cb5b25df2bff4c6, entries=150, sequenceid=292, filesize=11.9 K 2023-04-03 05:54:53,936 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,936 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:53,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:53,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:53,936 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,936 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:53,937 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/c6a92b601d3f4904ae46ae1470c56c8d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/c6a92b601d3f4904ae46ae1470c56c8d 2023-04-03 05:54:53,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:53,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:53,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:53,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:53,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:53,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:53,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:53,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:53,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:53,938 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:53,938 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:53,938 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:53,938 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:53,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:53,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:53,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:53,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:53,940 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:53,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,940 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:53,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/c6a92b601d3f4904ae46ae1470c56c8d, entries=150, sequenceid=292, filesize=11.9 K 2023-04-03 05:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,941 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 15471ccc940d59438142ba180c50365a in 85ms, sequenceid=292, compaction requested=false 2023-04-03 05:54:53,941 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:53,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:53,942 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42832368[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:53,942 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:53,942 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,943 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:53,944 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,944 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:53,944 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:53,945 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:53,946 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:53,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:53,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:53,948 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:53,948 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:53,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:53,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:53,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:53,949 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:53,951 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,951 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:53,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,952 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:53,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:53,952 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:53,953 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:53,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:53,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:53,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:53,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:53,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:53,957 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:53,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:53,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:53,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:53,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:53,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:53,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:53,957 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:53,957 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:53,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:53,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:53,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:53,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:53,991 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 05:54:53,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:53,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:53,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:53,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:53,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:53,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:54,046 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:54,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501354043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,058 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:54,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:54,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:54,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:54,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:54,063 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:54,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:54,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:54,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,065 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,065 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,065 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,065 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,066 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:54,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,066 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:54,067 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:54,067 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,067 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:54,068 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,068 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:54,068 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:54,068 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:54,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:54,073 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,073 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:54,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:54,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fb8607c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:54,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:54,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,077 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:54,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:54,077 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:54,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,078 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:54,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:54,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:54,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:54,082 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:54,082 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:54,082 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:54,082 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:54,082 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:54,083 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,084 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,094 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,094 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,094 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,094 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,094 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,094 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,094 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:54,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:54,094 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,095 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,095 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:54,095 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,095 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:54,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:54,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501354147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,195 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:54,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:54,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:54,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:54,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:54,199 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:54,199 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,199 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:54,199 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:54,199 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,199 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,200 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,200 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,201 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,201 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:54,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:54,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:54,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:54,202 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,202 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:54,202 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:54,202 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:54,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:54,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:54,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:54,204 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,204 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,205 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:54,206 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@426c58a0[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:54,206 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:54,206 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,208 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:54,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:54,208 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,208 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:54,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:54,210 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,210 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:54,210 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,210 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:54,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:54,212 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:54,212 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:54,212 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:54,212 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:54,213 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:54,214 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,214 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,214 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,214 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,216 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,218 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:54,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,218 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:54,218 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:54,218 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:54,218 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:54,252 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/bb0f5f8c0f5541838dd7c6f9c687bef4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/bb0f5f8c0f5541838dd7c6f9c687bef4 2023-04-03 05:54:54,256 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/A of 15471ccc940d59438142ba180c50365a into bb0f5f8c0f5541838dd7c6f9c687bef4(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:54,256 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:54,256 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/A, priority=13, startTime=1680501293822; duration=0sec 2023-04-03 05:54:54,256 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:54,300 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:54,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41216 deadline: 1680501354299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,302 DEBUG [Thread-1874] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:54:54,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:54,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41204 deadline: 1680501354302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,305 DEBUG [Thread-1872] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:54:54,319 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:54,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:54,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:54,322 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:54,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41168 deadline: 1680501354320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,323 DEBUG [Thread-1870] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8168 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:54:54,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:54,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:54,324 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:54,324 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,324 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:54,325 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:54,325 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,325 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,327 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,327 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:54,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:54,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:54,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:54,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:54,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:54,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:54,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:54,329 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,329 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:54,329 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:54,329 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:54,329 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:54,330 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,330 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:54,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41230 deadline: 1680501354328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:54,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,332 DEBUG [Thread-1876] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:54,333 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:54,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@366ff57b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:54,333 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,334 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,334 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:54,334 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:54,334 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:54,335 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:54,335 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,335 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,335 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:54,335 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:54,335 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:54,335 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:54,335 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:54,335 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:54,335 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:54,335 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:54,338 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,338 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,338 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,339 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,340 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,342 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:54,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:54,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:54,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,343 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:54,343 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:54,343 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,343 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,343 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:54,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501354349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,400 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/7856e0e19cf04e91b0bf6e573173c69f 2023-04-03 05:54:54,443 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:54,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:54,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:54,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:54,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:54,446 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:54,446 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,446 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:54,446 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:54,447 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,447 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,448 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,448 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,448 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,448 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:54,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:54,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:54,449 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:54,449 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,449 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:54,449 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:54,449 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:54,449 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,450 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:54,450 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:54,450 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:54,452 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,453 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:54,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d2c08bd[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:54,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:54,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,455 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,455 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,456 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:54,456 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:54,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,460 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,460 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,460 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:54,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:54,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:54,462 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:54,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:54,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:54,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:54,462 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:54,465 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,465 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,465 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:54,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:54,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,470 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,471 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,471 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:54,471 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,471 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:54,571 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:54,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:54,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:54,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:54,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:54,575 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:54,575 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,575 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:54,575 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:54,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,578 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:54,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:54,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:54,579 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,579 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:54,579 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:54,579 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:54,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,580 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:54,580 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:54,581 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,581 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,581 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:54,583 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:54,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2540a055[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:54,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:54,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:54,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,586 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:54,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:54,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:54,586 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,586 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:54,587 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:54,587 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,587 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:54,591 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:54,591 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:54,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:54,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:54,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:54,591 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:54,593 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,593 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,593 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,595 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:54,598 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:54,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,598 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,598 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:54,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:54,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,598 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:54,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:54,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501354654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,698 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:54,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:54,701 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:54,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:54,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:54,702 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:54,702 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,702 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:54,702 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:54,703 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,703 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,704 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,704 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,705 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,705 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:54,705 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,706 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:54,706 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:54,706 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,706 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:54,707 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,707 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:54,707 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:54,707 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:54,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:54,708 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,708 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:54,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@413b2ee4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:54,711 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:54,711 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,712 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,712 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:54,712 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:54,712 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:54,712 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,712 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:54,717 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:54,717 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,717 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,717 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:54,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:54,719 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:54,719 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:54,719 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:54,719 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:54,719 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:54,721 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,721 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,721 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,724 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:54,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:54,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,728 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,728 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:54,729 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:54,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,728 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,811 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/7d38191082b34331ac2594fb35db422a 2023-04-03 05:54:54,826 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/fe4c6700bf2547dca47e93d2c73b2d0a 2023-04-03 05:54:54,829 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:54,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:54,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:54,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:54,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:54,832 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:54,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:54,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:54,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,833 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,834 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/7856e0e19cf04e91b0bf6e573173c69f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7856e0e19cf04e91b0bf6e573173c69f 2023-04-03 05:54:54,834 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,834 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:54,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,835 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:54,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:54,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:54,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:54,835 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:54,836 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,836 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:54,836 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:54,836 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:54,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:54,838 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7856e0e19cf04e91b0bf6e573173c69f, entries=200, sequenceid=306, filesize=14.3 K 2023-04-03 05:54:54,839 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/7d38191082b34331ac2594fb35db422a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/7d38191082b34331ac2594fb35db422a 2023-04-03 05:54:54,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,842 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,842 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,842 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:54,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,843 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/7d38191082b34331ac2594fb35db422a, entries=150, sequenceid=306, filesize=11.9 K 2023-04-03 05:54:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,844 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/fe4c6700bf2547dca47e93d2c73b2d0a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/fe4c6700bf2547dca47e93d2c73b2d0a 2023-04-03 05:54:54,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:54,844 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61b2d73b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:54,844 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:54,844 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,845 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:54,846 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:54,846 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:54,846 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,846 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:54,847 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:54,847 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,847 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:54,847 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/fe4c6700bf2547dca47e93d2c73b2d0a, entries=150, sequenceid=306, filesize=11.9 K 2023-04-03 05:54:54,847 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,848 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 15471ccc940d59438142ba180c50365a in 857ms, sequenceid=306, compaction requested=true 2023-04-03 05:54:54,848 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,848 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:54,848 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:54,848 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:54,848 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:54,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,848 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,849 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:54,849 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/A is initiating minor compaction (all files) 2023-04-03 05:54:54,849 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/A in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,849 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/bb0f5f8c0f5541838dd7c6f9c687bef4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1d0cd8cabc37461e99af945f6c16fd63, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7856e0e19cf04e91b0bf6e573173c69f] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=38.6 K 2023-04-03 05:54:54,849 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:54,849 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/B is initiating minor compaction (all files) 2023-04-03 05:54:54,850 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/B in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,850 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e60664a5080c47109d7d6d1ae6377af0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6f3c249af15d4d199cb5b25df2bff4c6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/7d38191082b34331ac2594fb35db422a] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=36.2 K 2023-04-03 05:54:54,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,850 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting bb0f5f8c0f5541838dd7c6f9c687bef4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680501293049 2023-04-03 05:54:54,850 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e60664a5080c47109d7d6d1ae6377af0, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680501293049 2023-04-03 05:54:54,850 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 1d0cd8cabc37461e99af945f6c16fd63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680501293725 2023-04-03 05:54:54,851 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6f3c249af15d4d199cb5b25df2bff4c6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680501293725 2023-04-03 05:54:54,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,851 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 7856e0e19cf04e91b0bf6e573173c69f, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680501293869 2023-04-03 05:54:54,851 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 7d38191082b34331ac2594fb35db422a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680501293877 2023-04-03 05:54:54,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:54,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:54,852 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:54,852 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:54,852 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:54,852 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:54,852 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:54,857 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,857 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,857 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:54,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:54,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,858 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:54,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:54,860 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:54,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,863 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:54,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:54,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:54,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:54,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:54,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,864 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:54,864 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:54,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:54,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:54,865 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#A#compaction#292 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:54,865 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#B#compaction#293 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:54,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:54,880 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/eaefc4b8e69345e585211dfd0d1afc5a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/eaefc4b8e69345e585211dfd0d1afc5a 2023-04-03 05:54:54,884 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/B of 15471ccc940d59438142ba180c50365a into eaefc4b8e69345e585211dfd0d1afc5a(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:54,884 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:54,884 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/B, priority=13, startTime=1680501294848; duration=0sec 2023-04-03 05:54:54,884 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:54,884 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:54,885 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:54,885 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/C is initiating minor compaction (all files) 2023-04-03 05:54:54,886 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/C in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,886 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/7f51f28963974c7fa2df5a4d4bcc8833, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/c6a92b601d3f4904ae46ae1470c56c8d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/fe4c6700bf2547dca47e93d2c73b2d0a] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=36.2 K 2023-04-03 05:54:54,886 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 7f51f28963974c7fa2df5a4d4bcc8833, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=271, earliestPutTs=1680501293049 2023-04-03 05:54:54,886 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting c6a92b601d3f4904ae46ae1470c56c8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=292, earliestPutTs=1680501293725 2023-04-03 05:54:54,887 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting fe4c6700bf2547dca47e93d2c73b2d0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680501293877 2023-04-03 05:54:54,893 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#C#compaction#294 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:54,964 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:54,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:54,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:54,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:54,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:54,967 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:54,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:54,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:54,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:54,969 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:54,969 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:54,969 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:54,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:54,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:54,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:54,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:54,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:54,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:54,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:54,970 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:54,970 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:54,970 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:54,971 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 05:54:54,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:54,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:54,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:54,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:54,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:54,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:55,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:55,164 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:55,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:55,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501355196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,273 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/e55b39196eed41f09695295c6319a8c8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e55b39196eed41f09695295c6319a8c8 2023-04-03 05:54:55,278 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/A of 15471ccc940d59438142ba180c50365a into e55b39196eed41f09695295c6319a8c8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:55,278 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:55,278 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/A, priority=13, startTime=1680501294848; duration=0sec 2023-04-03 05:54:55,278 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:55,300 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:55,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501355299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,301 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/e58e110186f34df1962411dba4709fdf as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e58e110186f34df1962411dba4709fdf 2023-04-03 05:54:55,306 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/C of 15471ccc940d59438142ba180c50365a into e58e110186f34df1962411dba4709fdf(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:55,306 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:55,306 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/C, priority=13, startTime=1680501294848; duration=0sec 2023-04-03 05:54:55,306 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:55,387 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/19525e5e491c4158ae4ba99cb0f16528 2023-04-03 05:54:55,418 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/b030953182b54deeb150459047e272e2 2023-04-03 05:54:55,430 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=327 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/e6143a225f754a539e4241b66de853d3 2023-04-03 05:54:55,437 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/19525e5e491c4158ae4ba99cb0f16528 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/19525e5e491c4158ae4ba99cb0f16528 2023-04-03 05:54:55,441 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/19525e5e491c4158ae4ba99cb0f16528, entries=150, sequenceid=327, filesize=11.9 K 2023-04-03 05:54:55,442 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/b030953182b54deeb150459047e272e2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b030953182b54deeb150459047e272e2 2023-04-03 05:54:55,446 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b030953182b54deeb150459047e272e2, entries=150, sequenceid=327, filesize=11.9 K 2023-04-03 05:54:55,447 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/e6143a225f754a539e4241b66de853d3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e6143a225f754a539e4241b66de853d3 2023-04-03 05:54:55,450 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e6143a225f754a539e4241b66de853d3, entries=150, sequenceid=327, filesize=11.9 K 2023-04-03 05:54:55,451 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 15471ccc940d59438142ba180c50365a in 480ms, sequenceid=327, compaction requested=false 2023-04-03 05:54:55,451 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:55,451 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:55,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:55,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:55,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:55,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:55,453 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,453 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,454 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,454 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@283f325b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:55,456 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:55,456 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,457 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,457 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:55,457 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:55,457 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:55,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:55,459 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,459 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:55,459 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,459 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:55,462 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:55,462 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:55,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:55,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:55,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:55,462 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:55,469 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,469 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,469 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,471 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,474 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:55,474 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:55,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:55,474 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,474 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,474 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,474 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,474 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:55,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:55,474 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:55,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:55,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:55,474 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,474 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,475 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:55,475 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:55,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:55,505 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:54:55,505 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:55,505 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:55,505 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:55,505 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:55,505 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:55,505 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:55,524 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/deb5da4ff59b42db930c74db3d055b98 2023-04-03 05:54:55,568 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:55,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501355566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,575 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:55,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:55,577 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:55,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:55,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:55,581 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:55,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:55,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:55,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:55,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,583 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:55,583 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:55,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,585 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:55,588 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:55,588 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:55,588 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:55,589 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:55,589 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:55,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:55,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:55,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:55,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:55,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25dbbe01[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:55,600 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:55,600 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,605 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:55,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:55,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:55,605 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,605 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:55,607 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:55,607 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:55,610 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:55,610 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:55,610 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:55,610 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:55,610 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:55,612 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,612 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,612 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,614 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,617 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,617 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,617 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,617 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:55,617 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:55,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:55,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:55,617 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:55,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:55,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:55,617 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,617 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,617 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,617 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:55,617 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:55,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:55,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:55,618 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:55,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:55,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501355669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,718 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:55,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:55,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:55,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:55,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:55,721 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:55,721 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:55,721 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:55,721 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:55,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,723 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,723 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:55,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:55,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,723 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,723 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:55,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:55,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:55,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:55,724 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:55,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:55,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:55,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:55,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:55,725 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:55,725 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:55,725 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:55,725 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:55,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:55,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:55,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:55,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:55,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,727 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,727 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,727 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:55,729 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d0fe850[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:55,729 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:55,729 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,730 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:55,730 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,730 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:55,730 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:55,733 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,733 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:55,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:55,735 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:55,735 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:55,735 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:55,735 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:55,735 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:55,737 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,737 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:55,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,737 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:55,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,738 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,739 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,744 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:55,744 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:55,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:55,744 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:55,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:55,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:55,744 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,745 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,745 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,745 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,745 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,745 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,745 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:55,745 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:55,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:55,845 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:55,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:55,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:55,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:55,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:55,851 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:55,851 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:55,851 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:55,851 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:55,851 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,851 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,853 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:55,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:55,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:55,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,854 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:55,854 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:55,854 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:55,854 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:55,854 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:55,854 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:55,856 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:55,856 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:55,856 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:55,856 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:55,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:55,858 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,858 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,858 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,858 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:55,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4af3ebc5[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:55,861 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:55,861 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,862 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:55,863 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:55,863 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:55,863 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,863 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:55,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:55,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:55,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:55,867 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:55,867 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:55,867 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:55,867 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:55,867 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:55,869 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,869 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,869 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:55,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,871 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,873 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:55,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:55,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:55,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:55,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:55,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:55,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:55,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:55,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:55,876 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:55,876 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:55,879 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:55,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501355876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,968 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/634e649a6ef54b209534e502d2645d5c 2023-04-03 05:54:55,977 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:55,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:55,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:55,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:55,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:55,985 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:55,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:55,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:55,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:55,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,986 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,987 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:55,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:55,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:55,987 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=343 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/251a9e199f444182a61076065a92e6c5 2023-04-03 05:54:55,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:55,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:55,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:55,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:55,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:55,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:55,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:55,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:55,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:55,989 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:55,989 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:55,989 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:55,989 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:55,989 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:55,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,991 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,992 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/deb5da4ff59b42db930c74db3d055b98 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/deb5da4ff59b42db930c74db3d055b98 2023-04-03 05:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:55,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:55,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b8fb52a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:55,994 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:55,994 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,995 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:55,995 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:55,995 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,995 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:55,996 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/deb5da4ff59b42db930c74db3d055b98, entries=200, sequenceid=343, filesize=14.3 K 2023-04-03 05:54:55,997 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/634e649a6ef54b209534e502d2645d5c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/634e649a6ef54b209534e502d2645d5c 2023-04-03 05:54:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:55,997 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:55,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:56,000 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:56,000 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:56,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:56,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:56,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:56,000 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:56,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/634e649a6ef54b209534e502d2645d5c, entries=150, sequenceid=343, filesize=11.9 K 2023-04-03 05:54:56,002 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/251a9e199f444182a61076065a92e6c5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/251a9e199f444182a61076065a92e6c5 2023-04-03 05:54:56,003 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,003 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:56,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,003 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:56,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:56,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:56,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:56,004 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:56,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,007 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,007 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:56,007 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:56,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:56,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:56,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:56,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:56,007 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:56,007 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:56,007 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:56,008 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:56,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:56,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:56,008 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:56,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:56,008 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:56,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:56,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,011 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/251a9e199f444182a61076065a92e6c5, entries=150, sequenceid=343, filesize=11.9 K 2023-04-03 05:54:56,012 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 15471ccc940d59438142ba180c50365a in 507ms, sequenceid=343, compaction requested=true 2023-04-03 05:54:56,012 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:56,012 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:56,012 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:56,012 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:56,012 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:56,012 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:56,013 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:56,013 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/B is initiating minor compaction (all files) 2023-04-03 05:54:56,013 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/B in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:56,013 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/eaefc4b8e69345e585211dfd0d1afc5a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b030953182b54deeb150459047e272e2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/634e649a6ef54b209534e502d2645d5c] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=36.3 K 2023-04-03 05:54:56,014 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting eaefc4b8e69345e585211dfd0d1afc5a, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680501293877 2023-04-03 05:54:56,014 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b030953182b54deeb150459047e272e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680501294018 2023-04-03 05:54:56,014 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 634e649a6ef54b209534e502d2645d5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680501295176 2023-04-03 05:54:56,016 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:56,016 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/A is initiating minor compaction (all files) 2023-04-03 05:54:56,016 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/A in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:56,017 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e55b39196eed41f09695295c6319a8c8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/19525e5e491c4158ae4ba99cb0f16528, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/deb5da4ff59b42db930c74db3d055b98] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=38.7 K 2023-04-03 05:54:56,017 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e55b39196eed41f09695295c6319a8c8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680501293877 2023-04-03 05:54:56,017 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 19525e5e491c4158ae4ba99cb0f16528, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680501294018 2023-04-03 05:54:56,018 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting deb5da4ff59b42db930c74db3d055b98, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680501295176 2023-04-03 05:54:56,020 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#B#compaction#301 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:56,025 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#A#compaction#302 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:56,108 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:56,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:56,110 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:56,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:56,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:56,111 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:56,111 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:56,111 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:56,111 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:56,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:56,113 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:56,113 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:56,113 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,113 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:56,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:56,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:56,114 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:56,114 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:56,114 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:56,114 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:56,114 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:56,114 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:56,114 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:56,114 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:56,114 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:56,114 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:56,115 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 05:54:56,115 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:56,115 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:56,115 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:56,115 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:56,115 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:56,115 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:56,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:56,183 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:56,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:56,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501356218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:56,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501356320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,429 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/6e0a61c8a1cd46fc97ef349106c51e4b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6e0a61c8a1cd46fc97ef349106c51e4b 2023-04-03 05:54:56,432 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/aa3f029e4c614460881a88f5d0c4fe2e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/aa3f029e4c614460881a88f5d0c4fe2e 2023-04-03 05:54:56,434 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/B of 15471ccc940d59438142ba180c50365a into 6e0a61c8a1cd46fc97ef349106c51e4b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:56,434 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:56,434 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/B, priority=13, startTime=1680501296012; duration=0sec 2023-04-03 05:54:56,434 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:56,434 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:54:56,435 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:54:56,435 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/C is initiating minor compaction (all files) 2023-04-03 05:54:56,435 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/C in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:56,435 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e58e110186f34df1962411dba4709fdf, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e6143a225f754a539e4241b66de853d3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/251a9e199f444182a61076065a92e6c5] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=36.3 K 2023-04-03 05:54:56,436 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e58e110186f34df1962411dba4709fdf, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680501293877 2023-04-03 05:54:56,436 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e6143a225f754a539e4241b66de853d3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=327, earliestPutTs=1680501294018 2023-04-03 05:54:56,437 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 251a9e199f444182a61076065a92e6c5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680501295176 2023-04-03 05:54:56,437 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/A of 15471ccc940d59438142ba180c50365a into aa3f029e4c614460881a88f5d0c4fe2e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:56,437 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:56,437 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/A, priority=13, startTime=1680501296012; duration=0sec 2023-04-03 05:54:56,437 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:56,442 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#C#compaction#304 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:56,450 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/eaeebe2c127946b186d590bb7fe0b551 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/eaeebe2c127946b186d590bb7fe0b551 2023-04-03 05:54:56,454 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/C of 15471ccc940d59438142ba180c50365a into eaeebe2c127946b186d590bb7fe0b551(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:56,454 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:56,454 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/C, priority=13, startTime=1680501296012; duration=0sec 2023-04-03 05:54:56,454 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:56,523 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/b7200214113143bb9b9ceeb3d1da23cd 2023-04-03 05:54:56,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:56,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501356523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,535 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/622d2b2bae184f7eb7af72a13a4a6684 2023-04-03 05:54:56,545 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=361 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/37f993b5de664cfcb4cc23f03ad8d03b 2023-04-03 05:54:56,548 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/b7200214113143bb9b9ceeb3d1da23cd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b7200214113143bb9b9ceeb3d1da23cd 2023-04-03 05:54:56,552 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b7200214113143bb9b9ceeb3d1da23cd, entries=150, sequenceid=361, filesize=11.9 K 2023-04-03 05:54:56,553 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/622d2b2bae184f7eb7af72a13a4a6684 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/622d2b2bae184f7eb7af72a13a4a6684 2023-04-03 05:54:56,558 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/622d2b2bae184f7eb7af72a13a4a6684, entries=150, sequenceid=361, filesize=11.9 K 2023-04-03 05:54:56,558 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/37f993b5de664cfcb4cc23f03ad8d03b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/37f993b5de664cfcb4cc23f03ad8d03b 2023-04-03 05:54:56,562 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/37f993b5de664cfcb4cc23f03ad8d03b, entries=150, sequenceid=361, filesize=11.9 K 2023-04-03 05:54:56,563 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 15471ccc940d59438142ba180c50365a in 448ms, sequenceid=361, compaction requested=false 2023-04-03 05:54:56,563 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:56,563 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:56,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:56,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:56,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:56,563 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:56,565 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,565 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:56,565 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:56,565 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:56,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:56,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63dc2bec[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:56,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:56,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:56,568 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:56,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:56,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:56,568 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:56,569 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,569 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:56,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:56,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:56,570 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:56,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:56,574 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:56,574 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:56,574 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:56,575 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:56,575 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:56,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:56,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:56,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,578 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:56,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:56,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:56,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:56,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:56,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:56,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:56,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:56,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:56,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:56,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,581 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:56,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:56,582 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:56,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:56,682 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:56,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:56,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:56,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:56,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:56,685 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:56,685 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:56,685 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:56,685 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:56,685 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,685 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:56,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,694 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:56,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:56,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:56,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:56,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:56,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:56,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:56,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:56,696 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:56,696 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:56,696 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:54:56,696 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:56,696 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:56,696 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:56,696 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:56,696 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:56,696 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:56,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:56,827 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:56,870 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:56,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501356868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:56,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:56,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501356971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,106 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/8c7da63e68854ffc86738ed5bce3a4cb 2023-04-03 05:54:57,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:57,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501357175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,478 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:57,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501357478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,516 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/523dbd26eab54e409330f5dd8962406c 2023-04-03 05:54:57,525 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=378 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/b1bcc5a75d824e8989427ac8feb9d14b 2023-04-03 05:54:57,529 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/8c7da63e68854ffc86738ed5bce3a4cb as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/8c7da63e68854ffc86738ed5bce3a4cb 2023-04-03 05:54:57,534 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/8c7da63e68854ffc86738ed5bce3a4cb, entries=150, sequenceid=378, filesize=11.9 K 2023-04-03 05:54:57,535 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/523dbd26eab54e409330f5dd8962406c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/523dbd26eab54e409330f5dd8962406c 2023-04-03 05:54:57,539 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/523dbd26eab54e409330f5dd8962406c, entries=150, sequenceid=378, filesize=11.9 K 2023-04-03 05:54:57,539 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/b1bcc5a75d824e8989427ac8feb9d14b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/b1bcc5a75d824e8989427ac8feb9d14b 2023-04-03 05:54:57,544 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/b1bcc5a75d824e8989427ac8feb9d14b, entries=150, sequenceid=378, filesize=11.9 K 2023-04-03 05:54:57,545 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 15471ccc940d59438142ba180c50365a in 849ms, sequenceid=378, compaction requested=true 2023-04-03 05:54:57,545 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:57,545 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:57,545 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:57,547 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:57,547 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:57,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:57,548 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:57,548 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:57,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:57,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:57,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:57,550 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:57,550 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21d302af[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:57,550 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:57,552 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:57,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:57,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:57,552 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:57,552 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:57,552 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:57,552 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,552 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:57,554 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:57,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:57,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:57,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:57,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:57,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:57,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:57,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:57,558 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:57,558 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:57,558 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:57,558 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:57,558 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:57,560 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:57,560 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:57,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:57,560 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:57,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:57,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:57,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:57,561 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:57,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:57,563 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:57,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:57,572 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:57,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:57,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:57,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:57,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:57,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:57,573 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:57,573 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:57,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:57,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:57,573 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:57,573 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:57,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:57,673 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:57,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:57,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:57,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:57,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:57,676 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:57,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:57,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:57,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:57,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:57,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:57,678 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,678 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:57,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:57,679 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:57,679 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:57,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:57,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:57,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:57,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:57,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:57,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:57,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:57,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:57,681 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 05:54:57,681 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:57,681 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:57,681 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:57,681 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:57,681 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:57,681 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:57,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:57,984 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:58,016 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:58,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 377 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501358014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,093 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/7da0bd09b3c941f188dffe546b031920 2023-04-03 05:54:58,103 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/6c76374c43084c29be0c5b9c5ffd9254 2023-04-03 05:54:58,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:58,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 379 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501358117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:58,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501358321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,516 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=396 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/d58fa2a99ecb4da68373e7c6e91eaf67 2023-04-03 05:54:58,520 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/7da0bd09b3c941f188dffe546b031920 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7da0bd09b3c941f188dffe546b031920 2023-04-03 05:54:58,523 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7da0bd09b3c941f188dffe546b031920, entries=150, sequenceid=396, filesize=11.9 K 2023-04-03 05:54:58,524 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/6c76374c43084c29be0c5b9c5ffd9254 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6c76374c43084c29be0c5b9c5ffd9254 2023-04-03 05:54:58,527 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6c76374c43084c29be0c5b9c5ffd9254, entries=150, sequenceid=396, filesize=11.9 K 2023-04-03 05:54:58,528 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/d58fa2a99ecb4da68373e7c6e91eaf67 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d58fa2a99ecb4da68373e7c6e91eaf67 2023-04-03 05:54:58,531 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d58fa2a99ecb4da68373e7c6e91eaf67, entries=150, sequenceid=396, filesize=11.9 K 2023-04-03 05:54:58,532 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 15471ccc940d59438142ba180c50365a in 852ms, sequenceid=396, compaction requested=true 2023-04-03 05:54:58,532 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:58,532 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:58,532 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:58,532 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:58,532 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,532 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:58,532 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:58,533 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,533 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:58,534 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,534 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:58,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:58,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:58,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@742b6893[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:58,535 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:58,535 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,536 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:58,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:58,537 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:58,537 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,537 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:58,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:58,538 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:58,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:58,541 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:58,541 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:58,541 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:58,541 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:58,541 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:58,543 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,543 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:58,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:58,543 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:58,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:58,544 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,545 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,547 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,547 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:58,547 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:58,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:58,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:58,547 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:58,547 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:58,548 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:58,548 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:58,547 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,547 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:58,547 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:58,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:58,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:58,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:58,626 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 05:54:58,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:58,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:58,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:58,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:58,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:58,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:58,645 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/834e19a327f74cf8a1f9b21e95400c93 2023-04-03 05:54:58,648 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:58,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:58,650 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:58,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:58,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:58,652 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:58,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:58,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:58,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:58,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:58,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:58,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,655 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,655 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:58,655 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:58,655 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:58,655 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:58,655 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:58,655 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:58,655 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:58,655 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:58,656 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:58,656 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:58,656 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:58,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:58,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:58,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:58,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:58,660 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:58,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:58,660 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,661 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,661 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:58,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:58,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:58,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@126ba5d9[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:58,662 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:58,662 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,666 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:58,666 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:58,667 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:58,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:58,668 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:58,668 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:58,668 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,668 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:58,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:58,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:58,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:58,671 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:58,671 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:58,671 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:58,671 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:58,671 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:58,672 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/43018a99ae894aadbb5264d61588e53d 2023-04-03 05:54:58,673 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,673 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:58,673 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:58,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:58,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:58,673 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:58,675 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:58,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:58,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:58,680 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:58,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:58,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:58,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:58,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:58,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,681 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,681 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,681 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:58,681 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:58,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:58,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:58,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:58,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:58,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 398 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501358690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,781 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:58,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:58,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:58,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:58,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:58,784 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:58,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:58,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:58,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:58,785 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,785 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,787 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:58,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:58,787 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,787 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,787 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:58,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:58,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,788 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:58,788 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:58,788 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:58,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:58,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:58,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:58,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:58,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:58,788 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:58,789 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:58,789 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:58,789 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:58,789 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:58,789 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:58,789 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,789 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:58,789 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:58,794 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:58,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 400 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501358793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,798 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:58,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,799 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:58,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:58,800 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106a6f33[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:58,800 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:58,800 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,801 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,801 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:58,801 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:58,801 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:58,802 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,802 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:58,803 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:58,803 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:58,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:58,805 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:58,805 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:58,806 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:58,806 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:58,806 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:58,806 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:58,808 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,808 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:58,808 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:58,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:58,809 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:58,810 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,813 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,813 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:58,813 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:58,813 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:58,813 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,813 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:58,813 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,813 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,813 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:58,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:58,813 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:58,813 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:58,813 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:58,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:58,913 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:58,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:58,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:58,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:58,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:58,916 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:58,916 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:58,917 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:58,917 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:58,917 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,917 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,918 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,918 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:58,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:58,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:58,919 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,919 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:58,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:58,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:58,920 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:58,920 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:58,920 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:58,920 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:58,920 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:58,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:58,922 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:58,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:58,923 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,923 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:58,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:58,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2915c1eb[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:58,924 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:58,924 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,926 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,926 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:58,926 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:58,926 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,926 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:58,926 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:58,928 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:58,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:58,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:58,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:58,930 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:58,930 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:58,930 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:58,930 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:58,930 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:58,930 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:58,932 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,932 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,932 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:58,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:58,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:58,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:58,932 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:58,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,933 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:58,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,936 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,936 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:58,936 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,936 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:58,936 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:58,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:58,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:58,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:58,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:58,936 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:58,936 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,936 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:58,936 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:58,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:58,936 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:58,936 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:58,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:58,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:58,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:58,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:58,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 402 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501358995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,037 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:59,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:59,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:59,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:59,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:59,041 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:59,041 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:59,041 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:59,041 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:59,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,043 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:59,043 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:59,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,043 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,043 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:59,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:59,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:59,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:59,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:59,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:59,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:59,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:59,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:59,045 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:59,045 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:59,045 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:59,046 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:59,046 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:59,049 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,053 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,053 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:59,053 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:59,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:59,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:59,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:59,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:59,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6555baa[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:59,056 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:59,056 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,057 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,057 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:59,057 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:59,057 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:59,058 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,058 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:59,061 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:59,061 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:59,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:59,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:59,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:59,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:59,063 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:59,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:59,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:59,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:59,063 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:59,065 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,065 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:59,065 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:59,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:59,065 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,066 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,068 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,068 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,068 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,068 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,068 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,068 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:59,068 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:59,068 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,068 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:59,069 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:59,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:59,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:59,069 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:59,069 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:59,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:59,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:59,083 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/8394e8bc2eff4b06aca1af9d2c3b3c53 2023-04-03 05:54:59,088 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/834e19a327f74cf8a1f9b21e95400c93 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/834e19a327f74cf8a1f9b21e95400c93 2023-04-03 05:54:59,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/834e19a327f74cf8a1f9b21e95400c93, entries=200, sequenceid=411, filesize=14.3 K 2023-04-03 05:54:59,092 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/43018a99ae894aadbb5264d61588e53d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/43018a99ae894aadbb5264d61588e53d 2023-04-03 05:54:59,095 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/43018a99ae894aadbb5264d61588e53d, entries=150, sequenceid=411, filesize=11.9 K 2023-04-03 05:54:59,096 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/8394e8bc2eff4b06aca1af9d2c3b3c53 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8394e8bc2eff4b06aca1af9d2c3b3c53 2023-04-03 05:54:59,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8394e8bc2eff4b06aca1af9d2c3b3c53, entries=150, sequenceid=411, filesize=11.9 K 2023-04-03 05:54:59,100 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 15471ccc940d59438142ba180c50365a in 474ms, sequenceid=411, compaction requested=true 2023-04-03 05:54:59,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:59,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:59,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:59,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:59,100 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:54:59,100 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:54:59,101 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64120 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:54:59,101 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61680 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:54:59,101 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/A is initiating minor compaction (all files) 2023-04-03 05:54:59,102 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/B is initiating minor compaction (all files) 2023-04-03 05:54:59,102 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/A in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:59,102 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/B in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:59,102 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/aa3f029e4c614460881a88f5d0c4fe2e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b7200214113143bb9b9ceeb3d1da23cd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/8c7da63e68854ffc86738ed5bce3a4cb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7da0bd09b3c941f188dffe546b031920, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/834e19a327f74cf8a1f9b21e95400c93] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=62.6 K 2023-04-03 05:54:59,102 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6e0a61c8a1cd46fc97ef349106c51e4b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/622d2b2bae184f7eb7af72a13a4a6684, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/523dbd26eab54e409330f5dd8962406c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6c76374c43084c29be0c5b9c5ffd9254, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/43018a99ae894aadbb5264d61588e53d] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=60.2 K 2023-04-03 05:54:59,102 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting aa3f029e4c614460881a88f5d0c4fe2e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680501295176 2023-04-03 05:54:59,102 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6e0a61c8a1cd46fc97ef349106c51e4b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680501295176 2023-04-03 05:54:59,103 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting b7200214113143bb9b9ceeb3d1da23cd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680501295545 2023-04-03 05:54:59,103 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 622d2b2bae184f7eb7af72a13a4a6684, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680501295545 2023-04-03 05:54:59,103 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8c7da63e68854ffc86738ed5bce3a4cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1680501296212 2023-04-03 05:54:59,103 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 523dbd26eab54e409330f5dd8962406c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1680501296212 2023-04-03 05:54:59,103 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 7da0bd09b3c941f188dffe546b031920, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680501296857 2023-04-03 05:54:59,104 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6c76374c43084c29be0c5b9c5ffd9254, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680501296857 2023-04-03 05:54:59,104 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 834e19a327f74cf8a1f9b21e95400c93, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680501298004 2023-04-03 05:54:59,104 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 43018a99ae894aadbb5264d61588e53d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680501298009 2023-04-03 05:54:59,113 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#B#compaction#316 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:59,113 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#A#compaction#317 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:59,169 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:59,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:59,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:59,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:59,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:59,175 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:59,175 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:59,175 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:59,175 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:59,175 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,175 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,178 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,178 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:59,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:59,178 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,178 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,178 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:59,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:59,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:59,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:59,179 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:59,179 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:59,179 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:59,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:59,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:59,180 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:59,180 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:59,180 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 05:54:59,180 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:59,181 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:59,181 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:59,181 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:59,181 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:59,181 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:59,189 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/6ade5a6b25224d209903fbb04f8e81ca 2023-04-03 05:54:59,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:59,300 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:59,337 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:59,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 416 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501359335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,439 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:59,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501359438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,530 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/3188e4523e9448088696f6318b0d03c1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/3188e4523e9448088696f6318b0d03c1 2023-04-03 05:54:59,534 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 15471ccc940d59438142ba180c50365a/A of 15471ccc940d59438142ba180c50365a into 3188e4523e9448088696f6318b0d03c1(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:59,534 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:59,534 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/A, priority=11, startTime=1680501299100; duration=0sec 2023-04-03 05:54:59,534 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:54:59,534 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:54:59,535 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:54:59,536 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/C is initiating minor compaction (all files) 2023-04-03 05:54:59,536 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/C in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:59,536 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/eaeebe2c127946b186d590bb7fe0b551, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/37f993b5de664cfcb4cc23f03ad8d03b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/b1bcc5a75d824e8989427ac8feb9d14b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d58fa2a99ecb4da68373e7c6e91eaf67, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8394e8bc2eff4b06aca1af9d2c3b3c53] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=60.2 K 2023-04-03 05:54:59,536 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting eaeebe2c127946b186d590bb7fe0b551, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=343, earliestPutTs=1680501295176 2023-04-03 05:54:59,537 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 37f993b5de664cfcb4cc23f03ad8d03b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=361, earliestPutTs=1680501295545 2023-04-03 05:54:59,537 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting b1bcc5a75d824e8989427ac8feb9d14b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=378, earliestPutTs=1680501296212 2023-04-03 05:54:59,537 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting d58fa2a99ecb4da68373e7c6e91eaf67, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=396, earliestPutTs=1680501296857 2023-04-03 05:54:59,537 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8394e8bc2eff4b06aca1af9d2c3b3c53, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680501298009 2023-04-03 05:54:59,547 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#C#compaction#320 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:54:59,550 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/4adb5fe83ecc471488d1d7b74ce482b8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/4adb5fe83ecc471488d1d7b74ce482b8 2023-04-03 05:54:59,561 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 15471ccc940d59438142ba180c50365a/B of 15471ccc940d59438142ba180c50365a into 4adb5fe83ecc471488d1d7b74ce482b8(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:59,561 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:59,561 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/B, priority=11, startTime=1680501299100; duration=0sec 2023-04-03 05:54:59,561 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:54:59,599 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/3c3de150c65441cbae4ba3e9c9afb595 2023-04-03 05:54:59,612 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=428 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/c5824b44c97e4d0694942c7c5a2a0e40 2023-04-03 05:54:59,615 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/6ade5a6b25224d209903fbb04f8e81ca as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6ade5a6b25224d209903fbb04f8e81ca 2023-04-03 05:54:59,619 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6ade5a6b25224d209903fbb04f8e81ca, entries=150, sequenceid=428, filesize=11.9 K 2023-04-03 05:54:59,620 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/3c3de150c65441cbae4ba3e9c9afb595 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/3c3de150c65441cbae4ba3e9c9afb595 2023-04-03 05:54:59,624 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/3c3de150c65441cbae4ba3e9c9afb595, entries=150, sequenceid=428, filesize=11.9 K 2023-04-03 05:54:59,626 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/c5824b44c97e4d0694942c7c5a2a0e40 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/c5824b44c97e4d0694942c7c5a2a0e40 2023-04-03 05:54:59,630 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/c5824b44c97e4d0694942c7c5a2a0e40, entries=150, sequenceid=428, filesize=11.9 K 2023-04-03 05:54:59,631 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 15471ccc940d59438142ba180c50365a in 451ms, sequenceid=428, compaction requested=false 2023-04-03 05:54:59,631 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:59,631 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:59,631 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:59,632 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:59,632 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,632 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:59,632 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:59,633 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,634 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,634 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:59,636 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:59,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:59,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:59,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:59,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@495f73d4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:59,639 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:59,640 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,641 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,641 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:59,641 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:59,641 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,641 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:59,641 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:59,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:54:59,652 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 05:54:59,652 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:54:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:59,652 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:59,652 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:59,652 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:54:59,652 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,652 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:59,652 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:54:59,652 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:54:59,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:59,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:59,657 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:59,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:59,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:59,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:59,657 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:59,659 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,659 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:59,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,660 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:59,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:59,660 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:59,662 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,664 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,665 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:59,664 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:59,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:59,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:59,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:59,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:59,665 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:59,665 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:59,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:59,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:59,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:59,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 434 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501359699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,765 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:59,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:59,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:59,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:59,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:59,770 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:59,770 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:59,770 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:59,770 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:59,770 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,770 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,772 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:59,773 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:59,773 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,773 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:59,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:59,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:59,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:59,774 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:59,774 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:59,774 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:59,774 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:59,774 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:59,774 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:59,774 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:59,774 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:59,774 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:59,774 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:59,775 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:59,776 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:59,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:59,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a8f9e3b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:59,778 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:59,778 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,779 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:59,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:59,779 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,779 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:59,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:59,781 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:59,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:59,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:59,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:59,783 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:59,783 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:59,783 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:59,783 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:59,783 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:59,786 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,786 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:59,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,786 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:59,786 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:59,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:59,790 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:59,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:59,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:59,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:59,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:59,791 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:59,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:59,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,791 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,791 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:59,791 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:59,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:59,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:59,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:54:59,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 436 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501359802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,891 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:54:59,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:54:59,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:54:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:54:59,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:54:59,894 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:54:59,894 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:59,894 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:54:59,894 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:54:59,895 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,895 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,896 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,896 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:59,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:59,896 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,896 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:54:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:54:59,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:54:59,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:54:59,897 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:54:59,897 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:54:59,898 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:59,898 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:54:59,898 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:54:59,898 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:54:59,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:54:59,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,899 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,900 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:54:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:59,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:54:59,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d120c18[Count = 0] remaining members to acquire global barrier 2023-04-03 05:54:59,901 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:54:59,901 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,904 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:54:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:54:59,904 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,904 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:54:59,904 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:54:59,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,905 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:54:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:59,905 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:59,906 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:54:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:59,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:54:59,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:54:59,908 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:54:59,908 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:54:59,909 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:54:59,909 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:54:59,909 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:54:59,910 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,910 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,910 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:54:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:54:59,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:54:59,911 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:54:59,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,912 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:54:59,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:54:59,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:54:59,914 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:54:59,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:54:59,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:54:59,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:54:59,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:54:59,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:54:59,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:54:59,915 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:54:59,915 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:54:59,915 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,915 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:54:59,915 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:54:59,915 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:54:59,915 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:54:59,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:54:59,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:54:59,969 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/7edf80103f674678b88df2d8caa0ea74 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/7edf80103f674678b88df2d8caa0ea74 2023-04-03 05:54:59,974 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 15471ccc940d59438142ba180c50365a/C of 15471ccc940d59438142ba180c50365a into 7edf80103f674678b88df2d8caa0ea74(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:54:59,974 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:54:59,974 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/C, priority=11, startTime=1680501299100; duration=0sec 2023-04-03 05:54:59,974 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:00,006 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:00,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 438 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501360005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,015 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:00,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:00,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:00,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:00,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:00,018 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:00,018 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,018 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:00,018 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:00,018 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,019 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:00,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:00,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:00,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:00,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:00,022 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,022 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:00,022 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:00,022 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:00,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:00,024 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,024 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,024 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,024 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:00,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21368e1b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:00,025 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:00,025 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,027 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:00,027 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:00,027 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,027 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:00,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,028 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:00,028 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,028 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:00,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:00,031 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:00,031 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:00,031 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:00,031 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:00,031 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:00,040 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,040 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,040 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,041 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,044 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:00,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:00,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:00,044 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:00,044 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:00,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/bd3204e0e96d4e7ca54aef24e0afa25b 2023-04-03 05:55:00,145 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:00,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:00,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:00,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:00,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:00,148 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:00,148 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,148 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:00,148 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:00,148 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,148 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,150 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,150 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,150 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,150 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:00,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:00,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:00,151 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:00,151 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:00,152 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,152 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:00,152 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:00,152 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:00,152 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:00,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:00,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:00,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1345883a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:00,156 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:00,156 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,158 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,158 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:00,158 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:00,158 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:00,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:00,160 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:00,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:00,162 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:00,162 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:00,162 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:00,162 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:00,162 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:00,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,165 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,166 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,169 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,169 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,169 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,169 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,169 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,169 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,169 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,169 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,169 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:00,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:00,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,169 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:00,169 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:00,269 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:00,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:00,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:00,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:00,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:00,273 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:00,273 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,273 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:00,273 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:00,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,276 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,276 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,277 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,277 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:00,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:00,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:00,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:00,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:00,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:00,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:00,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,280 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,280 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,280 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:00,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65cdb22f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:00,282 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:00,282 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,283 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:00,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:00,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:00,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:00,284 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,284 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:00,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:00,287 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:00,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:00,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:00,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:00,287 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:00,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,289 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,290 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,292 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,293 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:00,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:00,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:00,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,293 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:00,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,293 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:00,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:00,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 440 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501360307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,393 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:00,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:00,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:00,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:00,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:00,397 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:00,397 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,397 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:00,397 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:00,397 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,397 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,398 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,399 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,399 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:00,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:00,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:00,400 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,400 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:00,400 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:00,400 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:00,402 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:00,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78221042[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:00,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:00,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,405 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:00,406 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,406 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:00,406 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:00,407 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,407 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,407 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,407 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:00,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:00,409 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:00,409 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:00,410 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:00,410 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:00,410 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:00,411 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,411 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,411 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,413 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,423 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,424 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,424 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,424 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,424 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,424 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:00,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:00,424 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,424 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,424 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,424 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,424 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:00,424 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:00,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,482 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/44d22358f7c74d60ae8883412be94a62 2023-04-03 05:55:00,524 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:00,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:00,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:00,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:00,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:00,527 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:00,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:00,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:00,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,528 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,529 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,529 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,529 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,529 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:00,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:00,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:00,530 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:00,530 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,530 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:00,531 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,531 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:00,531 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:00,531 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:00,531 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:00,532 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,532 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,532 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,532 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:00,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:00,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d710f0b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:00,534 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:00,534 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,535 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:00,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:00,535 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,535 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:00,537 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,537 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,537 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,538 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:00,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:00,540 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:00,540 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:00,540 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:00,540 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:00,540 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:00,541 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,542 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,542 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,542 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,543 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,546 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,546 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,546 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:00,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:00,546 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,546 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,546 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,546 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,546 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,546 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:00,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:00,546 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,646 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:00,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:00,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:00,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:00,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:00,650 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:00,650 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,650 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:00,650 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:00,651 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,651 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,652 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:00,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,653 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:00,654 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:00,654 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:00,654 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,654 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:00,654 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:00,654 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:00,654 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,655 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:00,655 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:00,657 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,657 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,657 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,657 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:00,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524a5ec7[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:00,658 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:00,658 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,660 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:00,660 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,660 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:00,660 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:00,661 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,661 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:00,662 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,662 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:00,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:00,664 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:00,664 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:00,664 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:00,664 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:00,664 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:00,666 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,666 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,666 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,668 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,672 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,672 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,672 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,672 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,672 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,672 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:00,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:00,672 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,672 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,672 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,672 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,672 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:00,672 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:00,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,773 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:00,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:00,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:00,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:00,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:00,776 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:00,776 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,776 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:00,776 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:00,776 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,776 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,777 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,777 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,777 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,777 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,778 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:00,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,778 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:00,779 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:00,779 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:00,779 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,779 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:00,779 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:00,779 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:00,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:00,781 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:00,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a9ec26b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:00,782 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:00,782 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,783 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:00,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:00,784 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:00,786 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:00,786 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,786 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,786 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:00,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:00,788 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:00,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:00,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:00,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:00,788 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:00,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,791 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,792 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,794 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,794 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,794 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,794 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,795 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:00,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:00,794 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,795 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,795 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,795 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,795 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,795 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:00,795 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:00,795 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:00,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:00,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 442 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501360810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,891 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/f555da1ce9bc43039d0d30609c4d6fd4 2023-04-03 05:55:00,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/bd3204e0e96d4e7ca54aef24e0afa25b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/bd3204e0e96d4e7ca54aef24e0afa25b 2023-04-03 05:55:00,895 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:00,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:00,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:00,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:00,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:00,898 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:00,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:00,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:00,899 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/bd3204e0e96d4e7ca54aef24e0afa25b, entries=200, sequenceid=446, filesize=14.3 K 2023-04-03 05:55:00,900 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,900 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/44d22358f7c74d60ae8883412be94a62 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/44d22358f7c74d60ae8883412be94a62 2023-04-03 05:55:00,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/44d22358f7c74d60ae8883412be94a62, entries=150, sequenceid=446, filesize=11.9 K 2023-04-03 05:55:00,904 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,904 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,904 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,904 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:00,904 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/f555da1ce9bc43039d0d30609c4d6fd4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/f555da1ce9bc43039d0d30609c4d6fd4 2023-04-03 05:55:00,905 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:00,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,905 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:00,906 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:00,906 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:00,906 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,906 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:00,906 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:00,906 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:00,906 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,907 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:00,907 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:00,908 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/f555da1ce9bc43039d0d30609c4d6fd4, entries=150, sequenceid=446, filesize=11.9 K 2023-04-03 05:55:00,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,908 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,909 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 15471ccc940d59438142ba180c50365a in 1256ms, sequenceid=446, compaction requested=true 2023-04-03 05:55:00,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:00,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:00,909 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:00,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,909 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,909 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:00,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:00,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,910 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:00,910 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:00,910 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/A is initiating minor compaction (all files) 2023-04-03 05:55:00,910 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/B is initiating minor compaction (all files) 2023-04-03 05:55:00,910 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/A in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,910 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/B in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,910 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/3188e4523e9448088696f6318b0d03c1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6ade5a6b25224d209903fbb04f8e81ca, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/bd3204e0e96d4e7ca54aef24e0afa25b] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=39.0 K 2023-04-03 05:55:00,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,910 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/4adb5fe83ecc471488d1d7b74ce482b8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/3c3de150c65441cbae4ba3e9c9afb595, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/44d22358f7c74d60ae8883412be94a62] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=36.6 K 2023-04-03 05:55:00,911 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 4adb5fe83ecc471488d1d7b74ce482b8, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680501298009 2023-04-03 05:55:00,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,911 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 3188e4523e9448088696f6318b0d03c1, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680501298009 2023-04-03 05:55:00,911 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3c3de150c65441cbae4ba3e9c9afb595, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1680501298629 2023-04-03 05:55:00,912 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 6ade5a6b25224d209903fbb04f8e81ca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1680501298629 2023-04-03 05:55:00,912 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 44d22358f7c74d60ae8883412be94a62, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680501299323 2023-04-03 05:55:00,912 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting bd3204e0e96d4e7ca54aef24e0afa25b, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680501299320 2023-04-03 05:55:00,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:00,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3256797a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:00,912 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:00,912 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,913 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,913 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:00,914 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,914 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:00,914 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:00,917 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,917 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:00,917 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,917 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,920 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#A#compaction#325 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:00,921 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#B#compaction#326 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:00,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:00,923 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:00,923 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:00,923 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:00,923 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:00,923 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:00,925 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,925 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:00,925 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,928 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:00,930 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:00,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,932 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:00,932 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:00,933 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:00,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:00,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:00,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:00,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:00,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:00,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:00,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:00,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:00,933 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:00,933 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:00,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:00,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:00,938 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/e6efb33f0a9b41fe801dc54e8e2a8b21 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e6efb33f0a9b41fe801dc54e8e2a8b21 2023-04-03 05:55:00,949 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/A of 15471ccc940d59438142ba180c50365a into e6efb33f0a9b41fe801dc54e8e2a8b21(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:00,949 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:00,949 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/A, priority=13, startTime=1680501300909; duration=0sec 2023-04-03 05:55:00,949 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:00,949 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:00,950 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:00,950 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/C is initiating minor compaction (all files) 2023-04-03 05:55:00,950 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/C in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:00,950 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/7edf80103f674678b88df2d8caa0ea74, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/c5824b44c97e4d0694942c7c5a2a0e40, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/f555da1ce9bc43039d0d30609c4d6fd4] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=36.6 K 2023-04-03 05:55:00,951 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 7edf80103f674678b88df2d8caa0ea74, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680501298009 2023-04-03 05:55:00,951 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting c5824b44c97e4d0694942c7c5a2a0e40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=428, earliestPutTs=1680501298629 2023-04-03 05:55:00,952 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting f555da1ce9bc43039d0d30609c4d6fd4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680501299323 2023-04-03 05:55:00,959 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#C#compaction#327 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:00,970 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/d4b17c3d3266421687ac9398147db5ad as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d4b17c3d3266421687ac9398147db5ad 2023-04-03 05:55:00,974 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/C of 15471ccc940d59438142ba180c50365a into d4b17c3d3266421687ac9398147db5ad(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:00,974 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:00,974 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/C, priority=13, startTime=1680501300909; duration=0sec 2023-04-03 05:55:00,974 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:01,033 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:01,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:01,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:01,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:01,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:01,037 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:01,037 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,037 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:01,037 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:01,038 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,038 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,039 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,039 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,040 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,040 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:01,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:01,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:01,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:01,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:01,041 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:01,042 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 05:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:55:01,042 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:01,053 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/90a299d3917d4e63b76dba53414acaf1 2023-04-03 05:55:01,062 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/3304e71d4ac6405598aaacefe3921395 2023-04-03 05:55:01,084 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/01666e1796004bf08cecebcc90bd8193 2023-04-03 05:55:01,094 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/90a299d3917d4e63b76dba53414acaf1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/90a299d3917d4e63b76dba53414acaf1 2023-04-03 05:55:01,098 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/90a299d3917d4e63b76dba53414acaf1, entries=150, sequenceid=465, filesize=11.9 K 2023-04-03 05:55:01,099 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/3304e71d4ac6405598aaacefe3921395 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/3304e71d4ac6405598aaacefe3921395 2023-04-03 05:55:01,111 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/3304e71d4ac6405598aaacefe3921395, entries=150, sequenceid=465, filesize=11.9 K 2023-04-03 05:55:01,111 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/01666e1796004bf08cecebcc90bd8193 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/01666e1796004bf08cecebcc90bd8193 2023-04-03 05:55:01,122 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/01666e1796004bf08cecebcc90bd8193, entries=150, sequenceid=465, filesize=11.9 K 2023-04-03 05:55:01,123 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for 15471ccc940d59438142ba180c50365a in 81ms, sequenceid=465, compaction requested=false 2023-04-03 05:55:01,123 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:01,124 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:01,124 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:01,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,127 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:01,129 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57ba316c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:01,129 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:01,129 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,130 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,131 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,131 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:01,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,131 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:01,131 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:01,131 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:01,132 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:01,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,132 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,132 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:01,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:01,135 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:01,136 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:01,136 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:01,136 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:01,136 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:01,137 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,137 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,137 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,138 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,139 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,142 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,142 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,142 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,142 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:01,142 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:01,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,143 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:01,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:01,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,143 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:01,243 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:01,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:01,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:01,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:01,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:01,247 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:01,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:01,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:01,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,248 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:01,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,250 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:01,250 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:01,250 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:01,251 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,251 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:01,251 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:01,251 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:01,251 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:01,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,253 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,253 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:01,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5706e13a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:01,254 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:01,254 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,255 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:01,256 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,256 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:01,258 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:01,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:01,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:01,261 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:01,261 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:01,261 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:01,261 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:01,261 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:01,262 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,262 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,263 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,263 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,265 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,267 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,267 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,267 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,267 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,267 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,267 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,267 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:01,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:01,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:01,267 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,267 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,267 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,267 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:01,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,267 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:01,332 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/20c850f3092d4d85a1919e9507de9c84 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/20c850f3092d4d85a1919e9507de9c84 2023-04-03 05:55:01,337 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/B of 15471ccc940d59438142ba180c50365a into 20c850f3092d4d85a1919e9507de9c84(size=12.9 K), total size for store is 24.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:01,337 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:01,337 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/B, priority=13, startTime=1680501300909; duration=0sec 2023-04-03 05:55:01,337 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:01,368 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:01,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:01,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:01,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:01,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:01,371 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:01,371 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,371 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:01,371 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:01,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,373 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,373 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,374 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,374 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:01,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:01,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:01,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:01,375 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:01,375 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,375 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:01,375 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:01,375 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:01,375 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,375 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:01,375 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:01,375 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,376 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:01,376 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:01,376 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,376 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:01,376 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:01,381 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,381 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,382 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:01,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ba7fc2b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:01,383 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:01,384 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,385 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:01,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:01,385 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:01,387 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,387 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:01,387 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,387 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:01,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:01,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:01,389 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:01,390 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:01,390 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:01,390 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:01,390 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:01,391 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,391 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,391 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,392 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,393 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,396 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,396 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,396 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,396 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,396 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,396 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,396 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:01,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:01,396 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,396 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,396 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,397 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:01,397 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,497 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:01,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:01,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:01,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:01,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:01,500 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:01,500 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,500 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:01,500 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:01,501 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,501 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,502 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,502 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,502 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:01,503 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:01,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,504 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:01,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:01,504 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:01,504 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,504 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:01,504 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:01,504 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:01,504 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,504 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:01,505 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:01,505 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,505 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:01,505 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:01,505 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,505 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:01,505 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:01,507 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,507 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:01,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:01,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3bc78c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:01,509 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:01,509 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,510 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,510 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:01,510 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:01,510 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:01,510 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,510 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:01,512 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:01,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:01,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:01,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:01,514 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:01,515 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:01,515 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:01,515 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:01,515 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:01,516 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,516 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,516 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,518 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,525 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,528 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,528 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,528 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,528 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,528 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,528 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,528 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,529 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,529 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,528 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:01,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:01,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,529 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:01,530 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:01,630 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:01,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:01,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:01,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:01,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:01,633 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:01,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:01,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:01,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,634 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,635 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:01,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:01,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:01,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:01,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:01,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:01,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:01,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:01,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:01,637 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,637 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:01,637 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:01,637 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:01,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:01,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:01,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:01,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,639 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:01,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c8ca8f6[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:01,641 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:01,641 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,642 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,642 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:01,642 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:01,642 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:01,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,643 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,644 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:01,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:01,646 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:01,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:01,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:01,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:01,646 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:01,648 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,648 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,648 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,649 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,650 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:01,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:01,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,653 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:01,653 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:01,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:01,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,654 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,654 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,754 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:01,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:01,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:01,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:01,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:01,758 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:01,758 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,758 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:01,758 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:01,758 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,758 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,759 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,759 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:01,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:01,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:01,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:01,764 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:01,764 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,764 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:01,766 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,766 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:01,766 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:01,766 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,766 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:01,767 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:01,768 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,768 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,768 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,768 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:01,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:01,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ce3896c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:01,770 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:01,770 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,771 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:01,771 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:01,771 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:01,771 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,772 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:01,776 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:01,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:01,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:01,779 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:01,779 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:01,779 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:01,779 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:01,779 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:01,780 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,780 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,780 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,781 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,782 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,784 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,784 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,784 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,784 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,785 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,785 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,785 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,785 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,785 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,785 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:01,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:01,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,785 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:01,785 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:01,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:55:01,825 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:55:01,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:55:01,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:01,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:55:01,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:01,825 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:55:01,825 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:01,875 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501301849, max timestamp=1680501301874Num uniques 859; 2023-04-03 05:55:01,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501301849, max timestamp=1680501301874Num uniques 859; 2023-04-03 05:55:01,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:55:01,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:55:01,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501301849, max timestamp=1680501301874Num uniques 859; 2023-04-03 05:55:01,876 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:55:01,885 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:01,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:01,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:01,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:01,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:01,890 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:01,890 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,890 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:01,890 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:01,891 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,891 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,892 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,893 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,893 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:01,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,894 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:01,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:01,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:01,894 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:01,894 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:01,894 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:01,895 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,895 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:01,895 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:01,895 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:01,895 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:01,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:01,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58e1787f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:01,899 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:01,900 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,901 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,901 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:01,901 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:01,901 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:01,902 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:01,902 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,902 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:01,902 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:01,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:01,905 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:01,905 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:01,905 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:01,905 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:01,905 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:01,907 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,907 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:01,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,907 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:01,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,907 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:01,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,909 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:01,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:01,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,913 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,913 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,913 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:01,913 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:01,913 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:01,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:01,913 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:01,913 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:01,913 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:01,913 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:01,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:01,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:01,913 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:01,913 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:01,914 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:01,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:01,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:01,933 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:01,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 485 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501361931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,014 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:02,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:02,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:02,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:02,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:02,017 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:02,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:02,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:02,018 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,018 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,019 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,019 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,019 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:02,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:02,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,020 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:02,020 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:02,020 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:02,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:02,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:02,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:02,021 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,021 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:02,021 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:02,021 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:02,021 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:02,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,024 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:02,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c42c348[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:02,025 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:02,025 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,026 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,026 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:02,026 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,026 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:02,026 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:02,026 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:02,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,029 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:02,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:02,031 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:02,031 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:02,031 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:02,031 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:02,031 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:02,032 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,032 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,033 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,033 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:02,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 487 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501362034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,035 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:02,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:02,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:02,037 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,038 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:02,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,038 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:02,038 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,038 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,038 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,138 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:02,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:02,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:02,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:02,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:02,141 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:02,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:02,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:02,142 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,142 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:02,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,144 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:02,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,144 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:02,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:02,144 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:02,144 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:02,144 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,144 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:02,145 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,145 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:02,145 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:02,145 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:02,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:02,147 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,147 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:02,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65101f2e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:02,149 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:02,149 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,150 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:02,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:02,150 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,150 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:02,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:02,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,152 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,152 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,152 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:02,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:02,154 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:02,154 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:02,154 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:02,154 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:02,155 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:02,160 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,160 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,160 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,160 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,161 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,164 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:02,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:02,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,164 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:02,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:02,164 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:02,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,232 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/b49dd651e5de4ace9400d21adc6c10f5 2023-04-03 05:55:02,238 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:02,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 489 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501362235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,265 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:02,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:02,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:02,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:02,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:02,268 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:02,268 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,268 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:02,268 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:02,268 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,268 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,269 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:02,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:02,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:02,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:02,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:02,271 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:02,271 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,271 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:02,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:02,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:02,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:02,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:02,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:02,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:02,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:02,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:02,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,280 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,280 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,280 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:02,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65384209[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:02,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:02,282 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,284 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,284 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:02,284 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:02,284 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:02,284 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,284 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:02,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:02,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:02,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:02,288 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:02,288 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:02,288 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:02,288 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:02,288 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:02,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,290 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,291 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:02,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:02,294 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,294 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:02,294 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:02,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,295 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,295 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,394 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:02,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:02,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:02,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:02,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:02,398 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:02,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:02,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:02,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,401 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:02,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:02,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:02,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:02,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:02,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:02,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:02,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:02,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:02,402 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,403 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:02,403 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:02,403 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,403 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:02,403 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:02,403 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,403 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:02,403 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:02,404 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,404 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,405 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,405 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:02,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36c9ec51[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:02,407 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:02,407 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,407 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,408 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,408 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:02,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:02,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:02,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:02,409 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,409 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:02,409 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,409 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:02,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:02,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:02,411 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:02,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:02,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:02,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:02,412 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:02,418 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,418 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,418 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,418 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,419 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,421 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,421 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,421 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:02,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:02,421 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:02,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,421 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,422 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:02,421 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,422 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:02,422 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,421 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,422 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,422 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,422 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,522 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:02,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:02,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:02,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:02,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:02,525 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:02,525 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,525 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:02,525 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:02,525 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,525 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,527 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:02,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,527 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:02,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:02,528 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:02,528 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:02,528 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:02,528 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,528 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:02,528 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:02,528 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:02,528 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,529 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:02,529 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:02,529 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:02,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:02,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:02,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:02,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,530 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,531 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,531 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:02,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cfb2143[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:02,532 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:02,532 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,534 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,534 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:02,535 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,535 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:02,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:02,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:02,537 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,537 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,537 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,537 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:02,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:02,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:02,539 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:02,539 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:02,539 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:02,539 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:02,539 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:02,541 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,541 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,541 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,542 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,543 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:02,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 491 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501362541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:02,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:02,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,548 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:02,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,548 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:02,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,549 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:02,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,649 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:02,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:02,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:02,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:02,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:02,652 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:02,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:02,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:02,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:02,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:02,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,658 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:02,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:02,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:02,658 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:02,658 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,658 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:02,658 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:02,658 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:02,658 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,658 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:02,658 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/5b132ec2d526436092d9e104128ba196 2023-04-03 05:55:02,658 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:02,659 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:02,659 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:02,660 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,661 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,661 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,661 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:02,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3977bc9d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:02,663 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:02,663 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,664 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,665 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:02,665 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:02,665 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:02,666 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,666 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:02,668 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,668 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:02,668 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,668 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:02,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:02,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:02,674 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:02,674 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:02,674 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:02,674 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:02,674 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:02,675 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,675 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,675 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,679 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,681 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,681 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,682 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,681 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:02,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:02,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,682 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,682 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,682 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,682 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,682 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:02,682 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:02,682 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,682 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,782 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:02,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:02,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:02,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:02,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:02,788 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:02,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:02,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:02,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,789 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,790 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,790 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,790 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:02,791 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:02,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,791 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:02,792 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:02,792 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:02,792 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,792 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:02,792 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:02,792 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:02,795 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,795 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,795 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,795 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:02,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ef889b0[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:02,797 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:02,797 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,798 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:02,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:02,798 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,798 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:02,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:02,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:02,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:02,803 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:02,803 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:02,803 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:02,803 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:02,803 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:02,803 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:02,804 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,804 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,804 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,805 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,806 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,809 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,809 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,809 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,809 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,809 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,809 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,809 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,809 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:02,809 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:02,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:02,810 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,810 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,810 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:02,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,810 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:02,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,911 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:02,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:02,913 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:02,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:02,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:02,915 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:02,915 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,915 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:02,915 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:02,916 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,916 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,917 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,917 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,917 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,918 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,918 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:02,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,918 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:02,918 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:02,918 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:02,918 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:02,918 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:02,918 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:02,919 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:02,919 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:02,919 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,919 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:02,919 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:02,919 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:02,919 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:02,919 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:02,919 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,919 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:02,919 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:02,920 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:02,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67aa455a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:02,922 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:02,922 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,924 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,924 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:02,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:02,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:02,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:02,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:02,932 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:02,932 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,932 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,932 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:02,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:02,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:02,935 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:02,935 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:02,935 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:02,936 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:02,936 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:02,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:02,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:02,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,938 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,939 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:02,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,944 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:02,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:02,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:02,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:02,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:02,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:02,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:02,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:02,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:02,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:02,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:02,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:02,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:02,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:02,946 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:02,946 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:03,046 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:03,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:03,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:03,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 493 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501363044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:03,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:03,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:03,049 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:03,049 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:03,049 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:03,049 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:03,049 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:03,049 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:03,052 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:03,052 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:03,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:03,052 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:03,052 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:03,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:03,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:03,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:03,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:03,053 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:03,053 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:03,053 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:03,053 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:03,053 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:03,053 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:03,053 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:03,053 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:03,053 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:03,053 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:03,053 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:03,054 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:03,054 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:03,054 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,054 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:03,054 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:03,055 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:03,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:03,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:03,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:03,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:03,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:03,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:03,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:03,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:03,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:03,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@625e91d5[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:03,058 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:03,058 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:03,060 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:03,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:03,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:03,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:03,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:03,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:03,060 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,060 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:03,061 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,061 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:03,061 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:03,061 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:03,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:03,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:03,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:03,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:03,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:03,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:03,063 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:03,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:03,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:03,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:03,063 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:03,066 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:03,066 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:03,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:03,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:03,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:03,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:03,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:03,066 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:03,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:03,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:03,066 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:03,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:03,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,068 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:03,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,070 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,070 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:03,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:03,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:03,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:03,070 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:03,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:03,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:03,070 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:03,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:03,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:03,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:03,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:03,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:03,071 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:03,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:03,071 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:03,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=481 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/66e82a33c0014df0bf907064a8a8e884 2023-04-03 05:55:03,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/b49dd651e5de4ace9400d21adc6c10f5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b49dd651e5de4ace9400d21adc6c10f5 2023-04-03 05:55:03,097 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b49dd651e5de4ace9400d21adc6c10f5, entries=200, sequenceid=481, filesize=14.3 K 2023-04-03 05:55:03,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/5b132ec2d526436092d9e104128ba196 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5b132ec2d526436092d9e104128ba196 2023-04-03 05:55:03,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5b132ec2d526436092d9e104128ba196, entries=150, sequenceid=481, filesize=11.9 K 2023-04-03 05:55:03,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/66e82a33c0014df0bf907064a8a8e884 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/66e82a33c0014df0bf907064a8a8e884 2023-04-03 05:55:03,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/66e82a33c0014df0bf907064a8a8e884, entries=150, sequenceid=481, filesize=11.9 K 2023-04-03 05:55:03,106 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 15471ccc940d59438142ba180c50365a in 1280ms, sequenceid=481, compaction requested=true 2023-04-03 05:55:03,106 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:03,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:03,106 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:03,106 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:03,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:03,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:03,107 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:03,107 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/A is initiating minor compaction (all files) 2023-04-03 05:55:03,107 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:03,107 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/A in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:03,107 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/B is initiating minor compaction (all files) 2023-04-03 05:55:03,107 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e6efb33f0a9b41fe801dc54e8e2a8b21, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/90a299d3917d4e63b76dba53414acaf1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b49dd651e5de4ace9400d21adc6c10f5] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=39.1 K 2023-04-03 05:55:03,107 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/B in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:03,107 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/20c850f3092d4d85a1919e9507de9c84, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/3304e71d4ac6405598aaacefe3921395, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5b132ec2d526436092d9e104128ba196] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=36.7 K 2023-04-03 05:55:03,108 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e6efb33f0a9b41fe801dc54e8e2a8b21, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680501299323 2023-04-03 05:55:03,108 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 20c850f3092d4d85a1919e9507de9c84, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680501299323 2023-04-03 05:55:03,108 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 90a299d3917d4e63b76dba53414acaf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680501299670 2023-04-03 05:55:03,108 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3304e71d4ac6405598aaacefe3921395, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680501299670 2023-04-03 05:55:03,109 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting b49dd651e5de4ace9400d21adc6c10f5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1680501301816 2023-04-03 05:55:03,109 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 5b132ec2d526436092d9e104128ba196, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1680501301818 2023-04-03 05:55:03,125 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#A#compaction#334 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:03,137 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#B#compaction#335 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:03,171 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:03,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:03,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:03,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:03,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:03,181 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:03,181 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:03,181 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:03,181 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:03,181 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:03,181 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:03,183 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/a499eef5257b4198b7dc371fb0762691 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/a499eef5257b4198b7dc371fb0762691 2023-04-03 05:55:03,183 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:03,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:03,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:03,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:03,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:03,184 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:03,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:03,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:03,185 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:03,185 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:03,189 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/A of 15471ccc940d59438142ba180c50365a into a499eef5257b4198b7dc371fb0762691(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:03,189 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:03,189 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/A, priority=13, startTime=1680501303106; duration=0sec 2023-04-03 05:55:03,190 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:03,190 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:03,191 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:03,192 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/C is initiating minor compaction (all files) 2023-04-03 05:55:03,192 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/C in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:03,192 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d4b17c3d3266421687ac9398147db5ad, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/01666e1796004bf08cecebcc90bd8193, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/66e82a33c0014df0bf907064a8a8e884] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=36.7 K 2023-04-03 05:55:03,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:03,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:03,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:03,192 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting d4b17c3d3266421687ac9398147db5ad, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680501299323 2023-04-03 05:55:03,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:03,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:03,193 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 01666e1796004bf08cecebcc90bd8193, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680501299670 2023-04-03 05:55:03,194 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 66e82a33c0014df0bf907064a8a8e884, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1680501301818 2023-04-03 05:55:03,194 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/a7821930fae74a1c886fa9cdf1fcd344 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/a7821930fae74a1c886fa9cdf1fcd344 2023-04-03 05:55:03,196 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:03,196 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:03,196 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:03,196 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:03,196 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 05:55:03,197 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:55:03,197 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:03,197 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:55:03,197 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:03,197 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:55:03,197 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:03,201 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#C#compaction#337 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:03,205 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/B of 15471ccc940d59438142ba180c50365a into a7821930fae74a1c886fa9cdf1fcd344(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:03,205 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:03,205 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/B, priority=13, startTime=1680501303106; duration=0sec 2023-04-03 05:55:03,205 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:03,217 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/64c5d5f016394f62b97e24bef9f31ed4 2023-04-03 05:55:03,612 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/be0d866f114a4d978fad533cd9ea85af as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/be0d866f114a4d978fad533cd9ea85af 2023-04-03 05:55:03,617 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/C of 15471ccc940d59438142ba180c50365a into be0d866f114a4d978fad533cd9ea85af(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:03,617 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:03,617 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/C, priority=13, startTime=1680501303106; duration=0sec 2023-04-03 05:55:03,617 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:03,627 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/e9c372708cd04f38a549e79979dff58d 2023-04-03 05:55:04,036 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=519 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/3222fc5ae5614b09ab17692be6fd3cb2 2023-04-03 05:55:04,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:55:04,050 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:04,055 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/64c5d5f016394f62b97e24bef9f31ed4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/64c5d5f016394f62b97e24bef9f31ed4 2023-04-03 05:55:04,059 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/64c5d5f016394f62b97e24bef9f31ed4, entries=150, sequenceid=519, filesize=11.9 K 2023-04-03 05:55:04,060 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/e9c372708cd04f38a549e79979dff58d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e9c372708cd04f38a549e79979dff58d 2023-04-03 05:55:04,063 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e9c372708cd04f38a549e79979dff58d, entries=150, sequenceid=519, filesize=11.9 K 2023-04-03 05:55:04,064 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/3222fc5ae5614b09ab17692be6fd3cb2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/3222fc5ae5614b09ab17692be6fd3cb2 2023-04-03 05:55:04,067 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/3222fc5ae5614b09ab17692be6fd3cb2, entries=150, sequenceid=519, filesize=11.9 K 2023-04-03 05:55:04,068 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=40.25 KB/41220 for 15471ccc940d59438142ba180c50365a in 871ms, sequenceid=519, compaction requested=false 2023-04-03 05:55:04,068 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:04,068 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:04,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:04,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:04,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:04,069 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,069 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,070 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,070 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 15471ccc940d59438142ba180c50365a 2023-04-03 05:55:04,070 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:55:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:55:04,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:04,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:55:04,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,070 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:55:04,070 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:04,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:04,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66c94a57[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:04,071 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:04,071 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,073 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,073 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:04,073 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:04,073 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:04,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:04,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:04,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:04,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:04,076 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:04,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:04,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:04,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:04,076 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:04,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,080 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,081 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:04,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:04,084 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:04,084 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:04,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,084 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,084 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:04,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501304104, max timestamp=1680501304111Num uniques 859; 2023-04-03 05:55:04,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501304104, max timestamp=1680501304111Num uniques 859; 2023-04-03 05:55:04,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:55:04,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501304104, max timestamp=1680501304111Num uniques 859; 2023-04-03 05:55:04,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:55:04,113 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:55:04,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:04,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 536 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501364173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,184 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:04,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:04,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:04,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:04,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:04,187 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:04,187 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,187 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:04,187 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:04,188 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,188 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,189 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:04,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:04,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,190 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:04,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:04,190 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:04,190 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,190 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:04,190 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:04,190 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:04,191 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,191 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:04,191 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:04,191 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,191 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:04,191 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:04,191 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,191 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:04,191 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:04,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,192 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:04,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:04,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ca6ea37[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:04,195 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:04,195 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,197 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,197 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:04,197 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,197 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:04,197 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:04,197 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:04,198 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:04,198 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,198 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:04,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:04,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:04,201 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:04,201 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:04,201 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:04,201 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:04,201 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:04,202 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,202 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,202 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,203 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,204 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,206 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,206 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:04,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:04,206 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:04,207 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:04,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,206 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,206 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,207 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:04,207 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,207 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,207 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,207 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,207 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:04,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 538 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501364276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,307 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:04,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:04,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:04,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:04,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:04,310 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:04,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:04,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:04,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,311 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,313 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,313 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,314 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,314 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:04,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:04,315 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:04,315 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:04,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:04,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:04,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:04,315 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:04,319 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,319 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,319 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,319 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:04,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@993f86c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:04,321 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:04,321 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,322 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,322 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:04,322 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,322 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:04,322 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:04,322 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:04,327 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,327 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,327 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,327 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:04,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:04,330 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:04,330 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:04,330 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:04,330 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:04,330 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:04,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:04,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41168 deadline: 1680501364330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,333 DEBUG [Thread-1870] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18178 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:55:04,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,334 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,335 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,337 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,337 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,337 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,337 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,337 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:04,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:04,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:04,337 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,338 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,338 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:04,338 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:04,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,338 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,338 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:04,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41204 deadline: 1680501364355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,358 DEBUG [Thread-1872] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18203 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:55:04,396 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:04,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41216 deadline: 1680501364395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,397 DEBUG [Thread-1874] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18242 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:55:04,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:04,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:41230 deadline: 1680501364431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,434 DEBUG [Thread-1876] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18281 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:55:04,438 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:04,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:04,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:04,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:04,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:04,441 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:04,441 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,441 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:04,441 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:04,441 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,441 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,443 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,443 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,443 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:04,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:04,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:04,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:04,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:04,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:04,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:04,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:04,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:04,445 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,445 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:04,445 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:04,445 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:04,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:04,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:04,445 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:04,447 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,447 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,448 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,448 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:04,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f5500b2[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:04,449 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:04,449 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:04,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:04,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:04,450 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,450 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:04,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,451 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:04,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:04,454 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:04,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:04,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:04,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:04,454 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:04,456 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,456 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,456 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,456 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,457 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:04,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:04,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,460 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,461 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:04,461 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:04,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,478 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/4d0dff18f45b4819b43fec4c57a28690 2023-04-03 05:55:04,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:04,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 540 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501364479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,561 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:04,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:04,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:04,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:04,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:04,564 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:04,564 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,564 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:04,564 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:04,564 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,565 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,566 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,566 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,566 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,566 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:04,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:04,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:04,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:04,567 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:04,567 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,567 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:04,567 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:04,567 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:04,567 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,568 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:04,568 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:04,568 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:04,568 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:04,569 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:04,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:04,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33627afd[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:04,571 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:04,571 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,572 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:04,572 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:04,572 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:04,572 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,572 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:04,574 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,574 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:04,574 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,574 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:04,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:04,576 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:04,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:04,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:04,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:04,577 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:04,578 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,578 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,578 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,579 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,580 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,582 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,582 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:04,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:04,582 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,582 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,583 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:04,583 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,583 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:04,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,583 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,583 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,583 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,583 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,683 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:04,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:04,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:04,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:04,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:04,687 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:04,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:04,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:04,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,688 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,688 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,689 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,689 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:04,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:04,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:04,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:04,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:04,690 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:04,690 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,690 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:04,690 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:04,690 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:04,690 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,691 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:04,691 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:04,691 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,691 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:04,691 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:04,691 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,691 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:04,691 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:04,693 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,693 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:04,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@389ed9f4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:04,695 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:04,695 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,699 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,699 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:04,699 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:04,699 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:04,704 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,704 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:04,705 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,705 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:04,705 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,705 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:04,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:04,712 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:04,712 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:04,712 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:04,712 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:04,712 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:04,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,716 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,717 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,721 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,721 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,721 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,721 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:04,721 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:04,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:04,721 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,721 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,721 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:04,722 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:04,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,721 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,722 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,722 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,722 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:04,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 542 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501364784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,822 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:04,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:04,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:04,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:04,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:04,825 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:04,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:04,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:04,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,826 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,827 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,827 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,827 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,827 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:04,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:04,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,828 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:04,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:04,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:04,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:04,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:04,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:04,829 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,829 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:04,829 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:04,829 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:04,829 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:04,830 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,830 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,831 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,831 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:04,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:04,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ed81613[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:04,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:04,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,834 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,834 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:04,834 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:04,834 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:04,834 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,834 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:04,835 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,835 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:04,835 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,835 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:04,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:04,838 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:04,838 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:04,838 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:04,838 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:04,838 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:04,839 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,839 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,839 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,840 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,841 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,843 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,843 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,843 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:04,843 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:04,843 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,843 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,843 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,843 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,843 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,843 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,843 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:04,843 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:04,843 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:04,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,888 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/b68aa0eb37e141ceb14247a258b5a35b 2023-04-03 05:55:04,944 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:04,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:04,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:04,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:04,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:04,948 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:04,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:04,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:04,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,950 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,950 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,951 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,951 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:04,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:04,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:04,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:04,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:04,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:04,952 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,952 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:04,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:04,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:04,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:04,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:04,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:04,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:04,954 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,954 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:04,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:04,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19bfd3ea[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:04,956 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:04,956 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:04,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:04,957 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,957 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:04,958 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,958 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:04,958 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,958 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:04,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:04,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:04,960 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:04,960 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:04,960 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:04,961 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:04,961 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:04,963 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,963 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,963 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:04,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,964 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,965 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:04,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:04,966 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,966 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:04,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:04,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:04,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:04,966 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:04,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:04,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:04,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:04,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:04,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:04,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:04,967 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:04,967 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:04,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:04,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:05,067 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:05,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:05,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:05,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:05,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:05,070 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:05,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:05,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:05,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:05,071 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,071 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,072 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:05,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:05,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:05,072 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,072 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:05,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:05,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,073 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:05,073 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:05,073 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:05,073 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:05,073 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:05,073 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:05,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:05,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:05,074 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:05,074 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:05,074 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:05,074 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:05,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:05,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:05,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:05,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:05,075 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:05,075 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:05,076 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,076 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:05,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:05,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:05,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:05,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:05,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@392af7b2[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:05,078 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:05,078 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,081 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,082 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:05,082 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:05,082 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:05,082 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,082 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:05,083 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:05,083 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,083 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:05,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:05,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:05,083 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:05,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:05,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:05,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:05,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:05,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:05,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:05,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:05,089 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:05,089 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:05,089 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:05,089 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:05,089 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:05,097 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,097 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:05,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,097 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:05,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:05,098 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:05,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:05,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:05,100 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:05,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,102 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,102 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,102 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,102 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:05,102 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:05,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:05,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:05,102 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:05,102 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:05,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:05,102 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:05,103 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:05,103 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:05,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:05,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:05,203 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:05,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:05,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:05,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:05,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:05,208 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:05,208 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:05,208 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:05,208 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:05,208 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,210 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,210 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:05,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:05,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:05,210 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,210 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:05,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:05,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:05,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:05,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:05,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:05,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:05,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:05,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:05,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:05,232 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:05,232 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:05,232 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. as already flushing 2023-04-03 05:55:05,232 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:05,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:05,235 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:05,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:05,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:05,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:05,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:05,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:05,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77b5d81a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:05,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:05,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,239 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:05,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:05,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:05,239 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,239 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:05,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:05,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:05,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:05,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:05,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:05,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:05,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:05,244 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:05,244 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:05,244 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:05,244 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:05,245 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:05,246 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,246 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,246 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:05,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:05,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:05,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:05,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:05,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:05,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:05,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,253 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,253 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:05,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:05,253 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:05,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:05,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:05,253 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:05,253 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:05,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:05,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:05,253 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:05,254 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:05,253 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,254 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,254 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,254 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:05,254 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,291 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:05,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 544 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:41208 deadline: 1680501365288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,301 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/142e5fb98ebf4e41951f2459e8111fc0 2023-04-03 05:55:05,305 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/4d0dff18f45b4819b43fec4c57a28690 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/4d0dff18f45b4819b43fec4c57a28690 2023-04-03 05:55:05,309 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/4d0dff18f45b4819b43fec4c57a28690, entries=150, sequenceid=531, filesize=11.9 K 2023-04-03 05:55:05,310 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/b68aa0eb37e141ceb14247a258b5a35b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b68aa0eb37e141ceb14247a258b5a35b 2023-04-03 05:55:05,315 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b68aa0eb37e141ceb14247a258b5a35b, entries=150, sequenceid=531, filesize=11.9 K 2023-04-03 05:55:05,315 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/142e5fb98ebf4e41951f2459e8111fc0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/142e5fb98ebf4e41951f2459e8111fc0 2023-04-03 05:55:05,319 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/142e5fb98ebf4e41951f2459e8111fc0, entries=150, sequenceid=531, filesize=11.9 K 2023-04-03 05:55:05,319 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 15471ccc940d59438142ba180c50365a in 1249ms, sequenceid=531, compaction requested=true 2023-04-03 05:55:05,320 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:05,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:05,320 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:05,320 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:05,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:05,320 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:05,321 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:05,321 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:05,321 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/B is initiating minor compaction (all files) 2023-04-03 05:55:05,321 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/A is initiating minor compaction (all files) 2023-04-03 05:55:05,321 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/B in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:05,321 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/A in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:05,321 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/a7821930fae74a1c886fa9cdf1fcd344, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e9c372708cd04f38a549e79979dff58d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b68aa0eb37e141ceb14247a258b5a35b] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=36.8 K 2023-04-03 05:55:05,321 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/a499eef5257b4198b7dc371fb0762691, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/64c5d5f016394f62b97e24bef9f31ed4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/4d0dff18f45b4819b43fec4c57a28690] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=36.8 K 2023-04-03 05:55:05,322 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a7821930fae74a1c886fa9cdf1fcd344, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1680501301818 2023-04-03 05:55:05,322 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting a499eef5257b4198b7dc371fb0762691, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1680501301818 2023-04-03 05:55:05,322 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e9c372708cd04f38a549e79979dff58d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1680501301923 2023-04-03 05:55:05,322 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 64c5d5f016394f62b97e24bef9f31ed4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1680501301923 2023-04-03 05:55:05,322 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b68aa0eb37e141ceb14247a258b5a35b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1680501304052 2023-04-03 05:55:05,323 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 4d0dff18f45b4819b43fec4c57a28690, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1680501304052 2023-04-03 05:55:05,332 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#A#compaction#343 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:05,340 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#B#compaction#344 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:05,354 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:05,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:05,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:05,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:05,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:05,357 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:05,357 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:05,357 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:05,357 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:05,357 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,357 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,358 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:05,358 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:05,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:05,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:05,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:05,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:05,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:05,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:05,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:05,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:05,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:05,360 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:05,360 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:05,360 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:05,360 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:05,360 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:05,360 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:05,360 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. started... 2023-04-03 05:55:05,360 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 05:55:05,360 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:55:05,360 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:05,361 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:55:05,361 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:05,361 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:55:05,361 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:05,369 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/083c527ccd564a99a7e9b8dcd03ac5f4 2023-04-03 05:55:05,750 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/ff26cc79f31448b5aa317de4f8c8990c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/ff26cc79f31448b5aa317de4f8c8990c 2023-04-03 05:55:05,750 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/a56faf4418c04cb9a86a5bfb4e244ddf as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/a56faf4418c04cb9a86a5bfb4e244ddf 2023-04-03 05:55:05,756 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/B of 15471ccc940d59438142ba180c50365a into a56faf4418c04cb9a86a5bfb4e244ddf(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:05,756 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:05,756 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/B, priority=13, startTime=1680501305320; duration=0sec 2023-04-03 05:55:05,756 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:05,756 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:05,757 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/A of 15471ccc940d59438142ba180c50365a into ff26cc79f31448b5aa317de4f8c8990c(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:05,757 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:05,757 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/A, priority=13, startTime=1680501305320; duration=0sec 2023-04-03 05:55:05,757 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:05,760 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:05,760 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 15471ccc940d59438142ba180c50365a/C is initiating minor compaction (all files) 2023-04-03 05:55:05,760 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 15471ccc940d59438142ba180c50365a/C in TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:05,760 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/be0d866f114a4d978fad533cd9ea85af, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/3222fc5ae5614b09ab17692be6fd3cb2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/142e5fb98ebf4e41951f2459e8111fc0] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp, totalSize=36.8 K 2023-04-03 05:55:05,761 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting be0d866f114a4d978fad533cd9ea85af, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=481, earliestPutTs=1680501301818 2023-04-03 05:55:05,761 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3222fc5ae5614b09ab17692be6fd3cb2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=519, earliestPutTs=1680501301923 2023-04-03 05:55:05,761 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 142e5fb98ebf4e41951f2459e8111fc0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1680501304052 2023-04-03 05:55:05,767 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 15471ccc940d59438142ba180c50365a#C#compaction#347 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:05,779 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/890ad8a62631408d9ab4e3d259a8192c 2023-04-03 05:55:05,999 DEBUG [Thread-1883] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x681ace6c to 127.0.0.1:63744 2023-04-03 05:55:06,000 DEBUG [Thread-1883] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:06,000 DEBUG [Thread-1887] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45de5ad8 to 127.0.0.1:63744 2023-04-03 05:55:06,000 DEBUG [Thread-1887] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:06,001 DEBUG [Thread-1881] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f29c1c4 to 127.0.0.1:63744 2023-04-03 05:55:06,001 DEBUG [Thread-1881] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:06,001 DEBUG [Thread-1879] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e5b1c14 to 127.0.0.1:63744 2023-04-03 05:55:06,001 DEBUG [Thread-1879] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:06,001 DEBUG [Thread-1885] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2b4fa5d3 to 127.0.0.1:63744 2023-04-03 05:55:06,001 DEBUG [Thread-1885] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:06,174 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/758894b44de44763ae196c18130e43fb as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/758894b44de44763ae196c18130e43fb 2023-04-03 05:55:06,177 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 15471ccc940d59438142ba180c50365a/C of 15471ccc940d59438142ba180c50365a into 758894b44de44763ae196c18130e43fb(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:06,178 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:06,178 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a., storeName=15471ccc940d59438142ba180c50365a/C, priority=13, startTime=1680501305320; duration=0sec 2023-04-03 05:55:06,178 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:06,188 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/b835bd8408d64fb19431839c699bd793 2023-04-03 05:55:06,191 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/083c527ccd564a99a7e9b8dcd03ac5f4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/083c527ccd564a99a7e9b8dcd03ac5f4 2023-04-03 05:55:06,194 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/083c527ccd564a99a7e9b8dcd03ac5f4, entries=150, sequenceid=568, filesize=11.9 K 2023-04-03 05:55:06,195 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/890ad8a62631408d9ab4e3d259a8192c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/890ad8a62631408d9ab4e3d259a8192c 2023-04-03 05:55:06,198 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/890ad8a62631408d9ab4e3d259a8192c, entries=150, sequenceid=568, filesize=11.9 K 2023-04-03 05:55:06,198 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/b835bd8408d64fb19431839c699bd793 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/b835bd8408d64fb19431839c699bd793 2023-04-03 05:55:06,201 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/b835bd8408d64fb19431839c699bd793, entries=150, sequenceid=568, filesize=11.9 K 2023-04-03 05:55:06,202 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 15471ccc940d59438142ba180c50365a in 842ms, sequenceid=568, compaction requested=false 2023-04-03 05:55:06,202 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:06,203 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:06,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:06,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:06,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:06,203 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:06,204 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:06,204 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:06,204 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:06,204 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:06,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:06,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19332071[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:06,205 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:06,205 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:06,206 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:06,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:06,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:06,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:06,207 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:06,207 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,207 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:06,209 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:06,209 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:06,209 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,209 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:06,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:06,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:06,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:06,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:06,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:06,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:06,211 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:06,211 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:06,211 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:06,211 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:06,211 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:06,212 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:06,212 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:06,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:06,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:06,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:06,212 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:06,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:06,213 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:06,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:06,214 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:06,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,215 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,215 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:06,215 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:06,215 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:06,215 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:06,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:06,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:06,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:06,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:06,215 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:06,216 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:06,216 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:06,216 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:06,216 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:06,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:06,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:06,216 DEBUG [Thread-1878] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:06,216 DEBUG [Thread-1878] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:06,294 DEBUG [Thread-1868] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x13544701 to 127.0.0.1:63744 2023-04-03 05:55:06,294 DEBUG [Thread-1868] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:06,316 DEBUG [Thread-1878] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:06,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:14,338 DEBUG [Thread-1870] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10b79c2c to 127.0.0.1:63744 2023-04-03 05:55:14,339 DEBUG [Thread-1870] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:14,424 DEBUG [Thread-1872] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5a5a81fa to 127.0.0.1:63744 2023-04-03 05:55:14,424 DEBUG [Thread-1872] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:14,447 DEBUG [Thread-1874] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3562c7bf to 127.0.0.1:63744 2023-04-03 05:55:14,447 DEBUG [Thread-1874] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:14,488 DEBUG [Thread-1876] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6705d115 to 127.0.0.1:63744 2023-04-03 05:55:14,488 DEBUG [Thread-1876] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 403 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 15 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 13 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 13 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 11 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 5678 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 5694 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 5683 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 5683 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 5670 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 05:55:14,489 INFO [Listener at localhost/37275] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 05:55:14,489 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x53921272 to 127.0.0.1:63744 2023-04-03 05:55:14,489 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:14,490 INFO [Listener at localhost/37275] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 05:55:14,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-03 05:55:14,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:14,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 05:55:14,494 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501314494"}]},"ts":"1680501314494"} 2023-04-03 05:55:14,495 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 05:55:14,497 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 05:55:14,497 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=15471ccc940d59438142ba180c50365a, UNASSIGN}] 2023-04-03 05:55:14,498 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=15471ccc940d59438142ba180c50365a, UNASSIGN 2023-04-03 05:55:14,499 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=15471ccc940d59438142ba180c50365a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:14,499 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501314499"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501314499"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501314499"}]},"ts":"1680501314499"} 2023-04-03 05:55:14,500 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:55:14,594 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 05:55:14,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 15471ccc940d59438142ba180c50365a 2023-04-03 05:55:14,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 15471ccc940d59438142ba180c50365a, disabling compactions & flushes 2023-04-03 05:55:14,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:14,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:14,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. after waiting 0 ms 2023-04-03 05:55:14,654 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:14,654 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 15471ccc940d59438142ba180c50365a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-03 05:55:14,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=A 2023-04-03 05:55:14,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:14,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=B 2023-04-03 05:55:14,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:14,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 15471ccc940d59438142ba180c50365a, store=C 2023-04-03 05:55:14,655 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:14,675 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/6e1f33908d1a4ee58da0b9e6200c9c3a 2023-04-03 05:55:14,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 05:55:15,089 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/6b321de79f1f4af4b45076d15452a746 2023-04-03 05:55:15,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 05:55:15,102 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/6d6e9c860caa44ad8774c9cffac4eb35 2023-04-03 05:55:15,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/A/6e1f33908d1a4ee58da0b9e6200c9c3a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6e1f33908d1a4ee58da0b9e6200c9c3a 2023-04-03 05:55:15,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6e1f33908d1a4ee58da0b9e6200c9c3a, entries=150, sequenceid=579, filesize=11.9 K 2023-04-03 05:55:15,109 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/B/6b321de79f1f4af4b45076d15452a746 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6b321de79f1f4af4b45076d15452a746 2023-04-03 05:55:15,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6b321de79f1f4af4b45076d15452a746, entries=150, sequenceid=579, filesize=11.9 K 2023-04-03 05:55:15,112 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/.tmp/C/6d6e9c860caa44ad8774c9cffac4eb35 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/6d6e9c860caa44ad8774c9cffac4eb35 2023-04-03 05:55:15,115 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/6d6e9c860caa44ad8774c9cffac4eb35, entries=150, sequenceid=579, filesize=11.9 K 2023-04-03 05:55:15,116 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 15471ccc940d59438142ba180c50365a in 462ms, sequenceid=579, compaction requested=true 2023-04-03 05:55:15,136 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/8bdb3adacdb34a868d91c0462c9443ef, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e01131a8fd03449a875fac34946b4822, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1c4b5aa539f44ca8b039bb62ebe3f81c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b50d6e9c4c86428fa2bcb45311c8fe3c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1cd37b76702e4710a9325348ce1fdc5a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/32855fa86a3e4bbebd8a52479d8d3e29, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6f7ed9cff7b5465abe258a1a8dfa7fd7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/f1540ce6a6834075bd65e101b5fc84cc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/2c8cb96d658447cda194342484d769ab, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/4652cf9b25234ee3abd18bbf61cc67c3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/19e65f306d7b442b9f4b57514471047d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/001718739bc047fab573e19a1e395b1a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/54b3980dafc945b19bf01d1d5cfd4f19, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7a655be4011f42279e740dd95c1ea3f1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e2546f94bf34435c95081a265a237511, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/40c96ae5a10f4c5f8af4c3d5fffdcc94, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/bb0f5f8c0f5541838dd7c6f9c687bef4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1d0cd8cabc37461e99af945f6c16fd63, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7856e0e19cf04e91b0bf6e573173c69f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e55b39196eed41f09695295c6319a8c8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/19525e5e491c4158ae4ba99cb0f16528, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/deb5da4ff59b42db930c74db3d055b98, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/aa3f029e4c614460881a88f5d0c4fe2e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b7200214113143bb9b9ceeb3d1da23cd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/8c7da63e68854ffc86738ed5bce3a4cb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7da0bd09b3c941f188dffe546b031920, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/834e19a327f74cf8a1f9b21e95400c93, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/3188e4523e9448088696f6318b0d03c1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6ade5a6b25224d209903fbb04f8e81ca, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/bd3204e0e96d4e7ca54aef24e0afa25b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e6efb33f0a9b41fe801dc54e8e2a8b21, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/90a299d3917d4e63b76dba53414acaf1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b49dd651e5de4ace9400d21adc6c10f5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/a499eef5257b4198b7dc371fb0762691, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/64c5d5f016394f62b97e24bef9f31ed4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/4d0dff18f45b4819b43fec4c57a28690] to archive 2023-04-03 05:55:15,137 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:55:15,139 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/8bdb3adacdb34a868d91c0462c9443ef to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/8bdb3adacdb34a868d91c0462c9443ef 2023-04-03 05:55:15,140 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e01131a8fd03449a875fac34946b4822 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e01131a8fd03449a875fac34946b4822 2023-04-03 05:55:15,141 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1c4b5aa539f44ca8b039bb62ebe3f81c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1c4b5aa539f44ca8b039bb62ebe3f81c 2023-04-03 05:55:15,141 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b50d6e9c4c86428fa2bcb45311c8fe3c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b50d6e9c4c86428fa2bcb45311c8fe3c 2023-04-03 05:55:15,142 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1cd37b76702e4710a9325348ce1fdc5a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1cd37b76702e4710a9325348ce1fdc5a 2023-04-03 05:55:15,143 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/32855fa86a3e4bbebd8a52479d8d3e29 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/32855fa86a3e4bbebd8a52479d8d3e29 2023-04-03 05:55:15,144 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6f7ed9cff7b5465abe258a1a8dfa7fd7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6f7ed9cff7b5465abe258a1a8dfa7fd7 2023-04-03 05:55:15,145 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/f1540ce6a6834075bd65e101b5fc84cc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/f1540ce6a6834075bd65e101b5fc84cc 2023-04-03 05:55:15,145 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/2c8cb96d658447cda194342484d769ab to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/2c8cb96d658447cda194342484d769ab 2023-04-03 05:55:15,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/4652cf9b25234ee3abd18bbf61cc67c3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/4652cf9b25234ee3abd18bbf61cc67c3 2023-04-03 05:55:15,147 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/19e65f306d7b442b9f4b57514471047d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/19e65f306d7b442b9f4b57514471047d 2023-04-03 05:55:15,148 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/001718739bc047fab573e19a1e395b1a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/001718739bc047fab573e19a1e395b1a 2023-04-03 05:55:15,149 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/54b3980dafc945b19bf01d1d5cfd4f19 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/54b3980dafc945b19bf01d1d5cfd4f19 2023-04-03 05:55:15,149 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7a655be4011f42279e740dd95c1ea3f1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7a655be4011f42279e740dd95c1ea3f1 2023-04-03 05:55:15,150 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e2546f94bf34435c95081a265a237511 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e2546f94bf34435c95081a265a237511 2023-04-03 05:55:15,151 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/40c96ae5a10f4c5f8af4c3d5fffdcc94 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/40c96ae5a10f4c5f8af4c3d5fffdcc94 2023-04-03 05:55:15,152 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/bb0f5f8c0f5541838dd7c6f9c687bef4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/bb0f5f8c0f5541838dd7c6f9c687bef4 2023-04-03 05:55:15,153 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1d0cd8cabc37461e99af945f6c16fd63 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/1d0cd8cabc37461e99af945f6c16fd63 2023-04-03 05:55:15,153 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7856e0e19cf04e91b0bf6e573173c69f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7856e0e19cf04e91b0bf6e573173c69f 2023-04-03 05:55:15,154 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e55b39196eed41f09695295c6319a8c8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e55b39196eed41f09695295c6319a8c8 2023-04-03 05:55:15,155 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/19525e5e491c4158ae4ba99cb0f16528 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/19525e5e491c4158ae4ba99cb0f16528 2023-04-03 05:55:15,156 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/deb5da4ff59b42db930c74db3d055b98 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/deb5da4ff59b42db930c74db3d055b98 2023-04-03 05:55:15,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/aa3f029e4c614460881a88f5d0c4fe2e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/aa3f029e4c614460881a88f5d0c4fe2e 2023-04-03 05:55:15,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b7200214113143bb9b9ceeb3d1da23cd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b7200214113143bb9b9ceeb3d1da23cd 2023-04-03 05:55:15,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/8c7da63e68854ffc86738ed5bce3a4cb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/8c7da63e68854ffc86738ed5bce3a4cb 2023-04-03 05:55:15,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7da0bd09b3c941f188dffe546b031920 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/7da0bd09b3c941f188dffe546b031920 2023-04-03 05:55:15,160 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/834e19a327f74cf8a1f9b21e95400c93 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/834e19a327f74cf8a1f9b21e95400c93 2023-04-03 05:55:15,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/3188e4523e9448088696f6318b0d03c1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/3188e4523e9448088696f6318b0d03c1 2023-04-03 05:55:15,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6ade5a6b25224d209903fbb04f8e81ca to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6ade5a6b25224d209903fbb04f8e81ca 2023-04-03 05:55:15,162 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/bd3204e0e96d4e7ca54aef24e0afa25b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/bd3204e0e96d4e7ca54aef24e0afa25b 2023-04-03 05:55:15,163 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e6efb33f0a9b41fe801dc54e8e2a8b21 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/e6efb33f0a9b41fe801dc54e8e2a8b21 2023-04-03 05:55:15,164 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/90a299d3917d4e63b76dba53414acaf1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/90a299d3917d4e63b76dba53414acaf1 2023-04-03 05:55:15,165 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b49dd651e5de4ace9400d21adc6c10f5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/b49dd651e5de4ace9400d21adc6c10f5 2023-04-03 05:55:15,166 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/a499eef5257b4198b7dc371fb0762691 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/a499eef5257b4198b7dc371fb0762691 2023-04-03 05:55:15,166 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/64c5d5f016394f62b97e24bef9f31ed4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/64c5d5f016394f62b97e24bef9f31ed4 2023-04-03 05:55:15,167 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/4d0dff18f45b4819b43fec4c57a28690 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/4d0dff18f45b4819b43fec4c57a28690 2023-04-03 05:55:15,170 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/f053bb06bb50476b8791898f4f9111b6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5fdb028eccd142f1a86c866689cd70d4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5bf6aefebc4b4e38a6addbf105641823, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b089e71a59484f258b91827057955adc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/212ccdba67cb44ba89e104271c914278, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e276fde792dc49d1b28e3a02ae1d59e0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/8011b9a1a58d427382b0029e9860598a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/82a401174f804377ad149730e94b2cab, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/0dca981c6d7c4620a2cd2187bc94ce07, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/c69ffd267ff24fb2be37cf88ed6aa567, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6f7423637b454e2a829720f65bbce824, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/29be80cc10ba4dc3a3ec027b67efcd43, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/85a4e289b3b74ca98e301f264842348e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b779dfe2ee2a41c08139751cfe46dfb3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e2f66c62d2cc4a689710076a6962ef9f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e60664a5080c47109d7d6d1ae6377af0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/42691ec04d354585b5e95287008bbc27, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6f3c249af15d4d199cb5b25df2bff4c6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/eaefc4b8e69345e585211dfd0d1afc5a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/7d38191082b34331ac2594fb35db422a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b030953182b54deeb150459047e272e2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6e0a61c8a1cd46fc97ef349106c51e4b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/634e649a6ef54b209534e502d2645d5c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/622d2b2bae184f7eb7af72a13a4a6684, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/523dbd26eab54e409330f5dd8962406c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6c76374c43084c29be0c5b9c5ffd9254, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/4adb5fe83ecc471488d1d7b74ce482b8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/43018a99ae894aadbb5264d61588e53d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/3c3de150c65441cbae4ba3e9c9afb595, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/20c850f3092d4d85a1919e9507de9c84, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/44d22358f7c74d60ae8883412be94a62, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/3304e71d4ac6405598aaacefe3921395, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/a7821930fae74a1c886fa9cdf1fcd344, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5b132ec2d526436092d9e104128ba196, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e9c372708cd04f38a549e79979dff58d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b68aa0eb37e141ceb14247a258b5a35b] to archive 2023-04-03 05:55:15,171 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:55:15,172 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/f053bb06bb50476b8791898f4f9111b6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/f053bb06bb50476b8791898f4f9111b6 2023-04-03 05:55:15,173 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5fdb028eccd142f1a86c866689cd70d4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5fdb028eccd142f1a86c866689cd70d4 2023-04-03 05:55:15,174 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5bf6aefebc4b4e38a6addbf105641823 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5bf6aefebc4b4e38a6addbf105641823 2023-04-03 05:55:15,174 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b089e71a59484f258b91827057955adc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b089e71a59484f258b91827057955adc 2023-04-03 05:55:15,175 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/212ccdba67cb44ba89e104271c914278 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/212ccdba67cb44ba89e104271c914278 2023-04-03 05:55:15,176 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e276fde792dc49d1b28e3a02ae1d59e0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e276fde792dc49d1b28e3a02ae1d59e0 2023-04-03 05:55:15,177 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/8011b9a1a58d427382b0029e9860598a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/8011b9a1a58d427382b0029e9860598a 2023-04-03 05:55:15,178 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/82a401174f804377ad149730e94b2cab to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/82a401174f804377ad149730e94b2cab 2023-04-03 05:55:15,178 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/0dca981c6d7c4620a2cd2187bc94ce07 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/0dca981c6d7c4620a2cd2187bc94ce07 2023-04-03 05:55:15,179 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/c69ffd267ff24fb2be37cf88ed6aa567 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/c69ffd267ff24fb2be37cf88ed6aa567 2023-04-03 05:55:15,180 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6f7423637b454e2a829720f65bbce824 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6f7423637b454e2a829720f65bbce824 2023-04-03 05:55:15,180 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/29be80cc10ba4dc3a3ec027b67efcd43 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/29be80cc10ba4dc3a3ec027b67efcd43 2023-04-03 05:55:15,181 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/85a4e289b3b74ca98e301f264842348e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/85a4e289b3b74ca98e301f264842348e 2023-04-03 05:55:15,182 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b779dfe2ee2a41c08139751cfe46dfb3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b779dfe2ee2a41c08139751cfe46dfb3 2023-04-03 05:55:15,183 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e2f66c62d2cc4a689710076a6962ef9f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e2f66c62d2cc4a689710076a6962ef9f 2023-04-03 05:55:15,183 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e60664a5080c47109d7d6d1ae6377af0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e60664a5080c47109d7d6d1ae6377af0 2023-04-03 05:55:15,184 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/42691ec04d354585b5e95287008bbc27 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/42691ec04d354585b5e95287008bbc27 2023-04-03 05:55:15,185 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6f3c249af15d4d199cb5b25df2bff4c6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6f3c249af15d4d199cb5b25df2bff4c6 2023-04-03 05:55:15,185 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/eaefc4b8e69345e585211dfd0d1afc5a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/eaefc4b8e69345e585211dfd0d1afc5a 2023-04-03 05:55:15,186 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/7d38191082b34331ac2594fb35db422a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/7d38191082b34331ac2594fb35db422a 2023-04-03 05:55:15,187 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b030953182b54deeb150459047e272e2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b030953182b54deeb150459047e272e2 2023-04-03 05:55:15,188 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6e0a61c8a1cd46fc97ef349106c51e4b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6e0a61c8a1cd46fc97ef349106c51e4b 2023-04-03 05:55:15,188 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/634e649a6ef54b209534e502d2645d5c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/634e649a6ef54b209534e502d2645d5c 2023-04-03 05:55:15,189 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/622d2b2bae184f7eb7af72a13a4a6684 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/622d2b2bae184f7eb7af72a13a4a6684 2023-04-03 05:55:15,190 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/523dbd26eab54e409330f5dd8962406c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/523dbd26eab54e409330f5dd8962406c 2023-04-03 05:55:15,191 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6c76374c43084c29be0c5b9c5ffd9254 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6c76374c43084c29be0c5b9c5ffd9254 2023-04-03 05:55:15,191 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/4adb5fe83ecc471488d1d7b74ce482b8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/4adb5fe83ecc471488d1d7b74ce482b8 2023-04-03 05:55:15,192 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/43018a99ae894aadbb5264d61588e53d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/43018a99ae894aadbb5264d61588e53d 2023-04-03 05:55:15,193 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/3c3de150c65441cbae4ba3e9c9afb595 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/3c3de150c65441cbae4ba3e9c9afb595 2023-04-03 05:55:15,194 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/20c850f3092d4d85a1919e9507de9c84 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/20c850f3092d4d85a1919e9507de9c84 2023-04-03 05:55:15,195 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/44d22358f7c74d60ae8883412be94a62 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/44d22358f7c74d60ae8883412be94a62 2023-04-03 05:55:15,196 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/3304e71d4ac6405598aaacefe3921395 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/3304e71d4ac6405598aaacefe3921395 2023-04-03 05:55:15,197 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/a7821930fae74a1c886fa9cdf1fcd344 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/a7821930fae74a1c886fa9cdf1fcd344 2023-04-03 05:55:15,197 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5b132ec2d526436092d9e104128ba196 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/5b132ec2d526436092d9e104128ba196 2023-04-03 05:55:15,198 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e9c372708cd04f38a549e79979dff58d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/e9c372708cd04f38a549e79979dff58d 2023-04-03 05:55:15,199 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b68aa0eb37e141ceb14247a258b5a35b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/b68aa0eb37e141ceb14247a258b5a35b 2023-04-03 05:55:15,202 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/1d59ba0f4a0c416eaaf3141b792edcd3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/845fa9b92b5b4b549510ea98fe80d9e2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/932ff9894e3a4c85be1dcc6db4e6c9d7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/ed5ac0ef6cd04a5ab1eba0a2c4193631, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/18f7e3d55177470286ccd0f423cf7edf, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/2632cf60a9ec46078fb4ae509baa50c9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d248ca2f1f984b63b49100f1830ed68c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e086fd95baaa4578b6b51bad7d8785d2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8a09213cb3c945b58f9a5c528a318320, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8db7387285de432682be11570b3e929a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/24ea684fecb64209b54e65807e827e8c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/4b13fc2d62cc478ea8e57045d372935d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/a3943087fd4a4c7493489e76ce9a33b5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/bf541ce05a08409f95b8513da2ef6e95, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/7f51f28963974c7fa2df5a4d4bcc8833, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/5028cfeb1abe4a28ba7850691d91c91e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/c6a92b601d3f4904ae46ae1470c56c8d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e58e110186f34df1962411dba4709fdf, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/fe4c6700bf2547dca47e93d2c73b2d0a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e6143a225f754a539e4241b66de853d3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/eaeebe2c127946b186d590bb7fe0b551, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/251a9e199f444182a61076065a92e6c5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/37f993b5de664cfcb4cc23f03ad8d03b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/b1bcc5a75d824e8989427ac8feb9d14b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d58fa2a99ecb4da68373e7c6e91eaf67, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/7edf80103f674678b88df2d8caa0ea74, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8394e8bc2eff4b06aca1af9d2c3b3c53, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/c5824b44c97e4d0694942c7c5a2a0e40, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d4b17c3d3266421687ac9398147db5ad, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/f555da1ce9bc43039d0d30609c4d6fd4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/01666e1796004bf08cecebcc90bd8193, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/be0d866f114a4d978fad533cd9ea85af, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/66e82a33c0014df0bf907064a8a8e884, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/3222fc5ae5614b09ab17692be6fd3cb2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/142e5fb98ebf4e41951f2459e8111fc0] to archive 2023-04-03 05:55:15,202 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:55:15,203 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/1d59ba0f4a0c416eaaf3141b792edcd3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/1d59ba0f4a0c416eaaf3141b792edcd3 2023-04-03 05:55:15,204 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/845fa9b92b5b4b549510ea98fe80d9e2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/845fa9b92b5b4b549510ea98fe80d9e2 2023-04-03 05:55:15,205 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/932ff9894e3a4c85be1dcc6db4e6c9d7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/932ff9894e3a4c85be1dcc6db4e6c9d7 2023-04-03 05:55:15,206 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/ed5ac0ef6cd04a5ab1eba0a2c4193631 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/ed5ac0ef6cd04a5ab1eba0a2c4193631 2023-04-03 05:55:15,207 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/18f7e3d55177470286ccd0f423cf7edf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/18f7e3d55177470286ccd0f423cf7edf 2023-04-03 05:55:15,208 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/2632cf60a9ec46078fb4ae509baa50c9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/2632cf60a9ec46078fb4ae509baa50c9 2023-04-03 05:55:15,209 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d248ca2f1f984b63b49100f1830ed68c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d248ca2f1f984b63b49100f1830ed68c 2023-04-03 05:55:15,209 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e086fd95baaa4578b6b51bad7d8785d2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e086fd95baaa4578b6b51bad7d8785d2 2023-04-03 05:55:15,210 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8a09213cb3c945b58f9a5c528a318320 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8a09213cb3c945b58f9a5c528a318320 2023-04-03 05:55:15,211 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8db7387285de432682be11570b3e929a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8db7387285de432682be11570b3e929a 2023-04-03 05:55:15,212 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/24ea684fecb64209b54e65807e827e8c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/24ea684fecb64209b54e65807e827e8c 2023-04-03 05:55:15,213 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/4b13fc2d62cc478ea8e57045d372935d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/4b13fc2d62cc478ea8e57045d372935d 2023-04-03 05:55:15,214 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/a3943087fd4a4c7493489e76ce9a33b5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/a3943087fd4a4c7493489e76ce9a33b5 2023-04-03 05:55:15,215 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/bf541ce05a08409f95b8513da2ef6e95 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/bf541ce05a08409f95b8513da2ef6e95 2023-04-03 05:55:15,215 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/7f51f28963974c7fa2df5a4d4bcc8833 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/7f51f28963974c7fa2df5a4d4bcc8833 2023-04-03 05:55:15,216 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/5028cfeb1abe4a28ba7850691d91c91e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/5028cfeb1abe4a28ba7850691d91c91e 2023-04-03 05:55:15,217 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/c6a92b601d3f4904ae46ae1470c56c8d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/c6a92b601d3f4904ae46ae1470c56c8d 2023-04-03 05:55:15,218 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e58e110186f34df1962411dba4709fdf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e58e110186f34df1962411dba4709fdf 2023-04-03 05:55:15,218 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/fe4c6700bf2547dca47e93d2c73b2d0a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/fe4c6700bf2547dca47e93d2c73b2d0a 2023-04-03 05:55:15,219 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e6143a225f754a539e4241b66de853d3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/e6143a225f754a539e4241b66de853d3 2023-04-03 05:55:15,219 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/eaeebe2c127946b186d590bb7fe0b551 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/eaeebe2c127946b186d590bb7fe0b551 2023-04-03 05:55:15,220 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/251a9e199f444182a61076065a92e6c5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/251a9e199f444182a61076065a92e6c5 2023-04-03 05:55:15,221 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/37f993b5de664cfcb4cc23f03ad8d03b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/37f993b5de664cfcb4cc23f03ad8d03b 2023-04-03 05:55:15,222 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/b1bcc5a75d824e8989427ac8feb9d14b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/b1bcc5a75d824e8989427ac8feb9d14b 2023-04-03 05:55:15,222 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d58fa2a99ecb4da68373e7c6e91eaf67 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d58fa2a99ecb4da68373e7c6e91eaf67 2023-04-03 05:55:15,223 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/7edf80103f674678b88df2d8caa0ea74 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/7edf80103f674678b88df2d8caa0ea74 2023-04-03 05:55:15,224 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8394e8bc2eff4b06aca1af9d2c3b3c53 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/8394e8bc2eff4b06aca1af9d2c3b3c53 2023-04-03 05:55:15,225 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/c5824b44c97e4d0694942c7c5a2a0e40 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/c5824b44c97e4d0694942c7c5a2a0e40 2023-04-03 05:55:15,226 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d4b17c3d3266421687ac9398147db5ad to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/d4b17c3d3266421687ac9398147db5ad 2023-04-03 05:55:15,226 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/f555da1ce9bc43039d0d30609c4d6fd4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/f555da1ce9bc43039d0d30609c4d6fd4 2023-04-03 05:55:15,227 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/01666e1796004bf08cecebcc90bd8193 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/01666e1796004bf08cecebcc90bd8193 2023-04-03 05:55:15,228 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/be0d866f114a4d978fad533cd9ea85af to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/be0d866f114a4d978fad533cd9ea85af 2023-04-03 05:55:15,228 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/66e82a33c0014df0bf907064a8a8e884 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/66e82a33c0014df0bf907064a8a8e884 2023-04-03 05:55:15,229 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/3222fc5ae5614b09ab17692be6fd3cb2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/3222fc5ae5614b09ab17692be6fd3cb2 2023-04-03 05:55:15,230 DEBUG [StoreCloser-TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/142e5fb98ebf4e41951f2459e8111fc0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/142e5fb98ebf4e41951f2459e8111fc0 2023-04-03 05:55:15,233 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/recovered.edits/582.seqid, newMaxSeqId=582, maxSeqId=1 2023-04-03 05:55:15,234 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a. 2023-04-03 05:55:15,234 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 15471ccc940d59438142ba180c50365a: 2023-04-03 05:55:15,236 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 15471ccc940d59438142ba180c50365a 2023-04-03 05:55:15,236 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=15471ccc940d59438142ba180c50365a, regionState=CLOSED 2023-04-03 05:55:15,236 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501315236"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501315236"}]},"ts":"1680501315236"} 2023-04-03 05:55:15,238 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-03 05:55:15,238 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 15471ccc940d59438142ba180c50365a, server=jenkins-hbase4.apache.org,37397,1680501214665 in 737 msec 2023-04-03 05:55:15,239 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-03 05:55:15,239 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=15471ccc940d59438142ba180c50365a, UNASSIGN in 741 msec 2023-04-03 05:55:15,239 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501315239"}]},"ts":"1680501315239"} 2023-04-03 05:55:15,240 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 05:55:15,241 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 05:55:15,242 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 752 msec 2023-04-03 05:55:15,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-03 05:55:15,597 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-03 05:55:15,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-03 05:55:15,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:15,599 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:15,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-03 05:55:15,600 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:15,605 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a 2023-04-03 05:55:15,607 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/recovered.edits] 2023-04-03 05:55:15,609 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/083c527ccd564a99a7e9b8dcd03ac5f4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/083c527ccd564a99a7e9b8dcd03ac5f4 2023-04-03 05:55:15,610 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6e1f33908d1a4ee58da0b9e6200c9c3a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/6e1f33908d1a4ee58da0b9e6200c9c3a 2023-04-03 05:55:15,611 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/ff26cc79f31448b5aa317de4f8c8990c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/A/ff26cc79f31448b5aa317de4f8c8990c 2023-04-03 05:55:15,613 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6b321de79f1f4af4b45076d15452a746 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/6b321de79f1f4af4b45076d15452a746 2023-04-03 05:55:15,614 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/890ad8a62631408d9ab4e3d259a8192c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/890ad8a62631408d9ab4e3d259a8192c 2023-04-03 05:55:15,615 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/a56faf4418c04cb9a86a5bfb4e244ddf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/B/a56faf4418c04cb9a86a5bfb4e244ddf 2023-04-03 05:55:15,617 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/6d6e9c860caa44ad8774c9cffac4eb35 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/6d6e9c860caa44ad8774c9cffac4eb35 2023-04-03 05:55:15,618 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/758894b44de44763ae196c18130e43fb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/758894b44de44763ae196c18130e43fb 2023-04-03 05:55:15,619 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/b835bd8408d64fb19431839c699bd793 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/C/b835bd8408d64fb19431839c699bd793 2023-04-03 05:55:15,621 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/recovered.edits/582.seqid to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a/recovered.edits/582.seqid 2023-04-03 05:55:15,621 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/15471ccc940d59438142ba180c50365a 2023-04-03 05:55:15,621 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 05:55:15,623 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:15,624 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 05:55:15,625 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 05:55:15,626 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:15,626 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 05:55:15,626 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680501315626"}]},"ts":"9223372036854775807"} 2023-04-03 05:55:15,627 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 05:55:15,627 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 15471ccc940d59438142ba180c50365a, NAME => 'TestAcidGuarantees,,1680501283829.15471ccc940d59438142ba180c50365a.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 05:55:15,627 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 05:55:15,627 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680501315627"}]},"ts":"9223372036854775807"} 2023-04-03 05:55:15,628 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 05:55:15,630 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:15,631 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-03 05:55:15,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-03 05:55:15,701 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-03 05:55:15,713 INFO [Listener at localhost/37275] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224 (was 224), OpenFileDescriptor=525 (was 513) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=318 (was 336), ProcessCount=169 (was 169), AvailableMemoryMB=3388 (was 3579) 2023-04-03 05:55:15,723 INFO [Listener at localhost/37275] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=224, OpenFileDescriptor=525, MaxFileDescriptor=60000, SystemLoadAverage=318, ProcessCount=169, AvailableMemoryMB=3387 2023-04-03 05:55:15,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 05:55:15,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 05:55:15,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:15,727 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 05:55:15,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-03 05:55:15,728 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 05:55:15,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 05:55:15,729 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:15,729 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb empty. 2023-04-03 05:55:15,730 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:15,730 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 05:55:15,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 05:55:16,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 05:55:16,136 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 05:55:16,138 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => c534d6cb3070e98a48ea840ac1a7b2fb, NAME => 'TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp 2023-04-03 05:55:16,142 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:55:16,143 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing c534d6cb3070e98a48ea840ac1a7b2fb, disabling compactions & flushes 2023-04-03 05:55:16,143 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:16,143 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:16,143 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. after waiting 0 ms 2023-04-03 05:55:16,143 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:16,143 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:16,143 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:16,145 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 05:55:16,146 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501316145"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501316145"}]},"ts":"1680501316145"} 2023-04-03 05:55:16,147 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 05:55:16,147 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 05:55:16,147 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501316147"}]},"ts":"1680501316147"} 2023-04-03 05:55:16,148 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 05:55:16,155 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c534d6cb3070e98a48ea840ac1a7b2fb, ASSIGN}] 2023-04-03 05:55:16,156 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c534d6cb3070e98a48ea840ac1a7b2fb, ASSIGN 2023-04-03 05:55:16,157 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c534d6cb3070e98a48ea840ac1a7b2fb, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37397,1680501214665; forceNewPlan=false, retain=false 2023-04-03 05:55:16,308 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=c534d6cb3070e98a48ea840ac1a7b2fb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:16,308 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501316308"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501316308"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501316308"}]},"ts":"1680501316308"} 2023-04-03 05:55:16,309 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:55:16,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 05:55:16,464 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:16,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c534d6cb3070e98a48ea840ac1a7b2fb, NAME => 'TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.', STARTKEY => '', ENDKEY => ''} 2023-04-03 05:55:16,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:16,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:55:16,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:16,464 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:16,465 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:16,466 DEBUG [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A 2023-04-03 05:55:16,466 DEBUG [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A 2023-04-03 05:55:16,466 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:55:16,467 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c534d6cb3070e98a48ea840ac1a7b2fb columnFamilyName A 2023-04-03 05:55:16,467 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.HStore(310): Store=c534d6cb3070e98a48ea840ac1a7b2fb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:55:16,467 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:16,468 DEBUG [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B 2023-04-03 05:55:16,468 DEBUG [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B 2023-04-03 05:55:16,468 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:55:16,468 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c534d6cb3070e98a48ea840ac1a7b2fb columnFamilyName B 2023-04-03 05:55:16,469 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.HStore(310): Store=c534d6cb3070e98a48ea840ac1a7b2fb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:55:16,469 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:16,470 DEBUG [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C 2023-04-03 05:55:16,470 DEBUG [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C 2023-04-03 05:55:16,470 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:55:16,470 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c534d6cb3070e98a48ea840ac1a7b2fb columnFamilyName C 2023-04-03 05:55:16,470 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.HStore(310): Store=c534d6cb3070e98a48ea840ac1a7b2fb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:55:16,470 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:16,471 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:16,471 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:16,472 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 05:55:16,473 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:16,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 05:55:16,475 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c534d6cb3070e98a48ea840ac1a7b2fb; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65299584, jitterRate=-0.026960372924804688}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 05:55:16,475 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:16,476 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., pid=37, masterSystemTime=1680501316461 2023-04-03 05:55:16,477 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:16,477 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:16,477 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=c534d6cb3070e98a48ea840ac1a7b2fb, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:16,477 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501316477"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680501316477"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680501316477"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680501316477"}]},"ts":"1680501316477"} 2023-04-03 05:55:16,479 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-03 05:55:16,479 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 in 169 msec 2023-04-03 05:55:16,480 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-03 05:55:16,480 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c534d6cb3070e98a48ea840ac1a7b2fb, ASSIGN in 324 msec 2023-04-03 05:55:16,481 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 05:55:16,481 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501316481"}]},"ts":"1680501316481"} 2023-04-03 05:55:16,482 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 05:55:16,483 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 05:55:16,484 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 758 msec 2023-04-03 05:55:16,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-03 05:55:16,832 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-03 05:55:16,832 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x67b6d133 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:16,837 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@494529f9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:16,838 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:16,841 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36572, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:16,841 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 05:55:16,842 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:54298, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 05:55:16,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 05:55:16,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 05:55:16,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:17,256 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 05:55:17,257 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 05:55:17,257 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 05:55:17,259 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-03 05:55:17,261 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c534d6cb3070e98a48ea840ac1a7b2fb, REOPEN/MOVE}] 2023-04-03 05:55:17,262 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c534d6cb3070e98a48ea840ac1a7b2fb, REOPEN/MOVE 2023-04-03 05:55:17,262 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c534d6cb3070e98a48ea840ac1a7b2fb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,262 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501317262"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501317262"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501317262"}]},"ts":"1680501317262"} 2023-04-03 05:55:17,263 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:55:17,415 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:17,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c534d6cb3070e98a48ea840ac1a7b2fb, disabling compactions & flushes 2023-04-03 05:55:17,416 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:17,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:17,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. after waiting 0 ms 2023-04-03 05:55:17,416 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:17,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-03 05:55:17,419 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:17,419 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:17,419 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: c534d6cb3070e98a48ea840ac1a7b2fb to self. 2023-04-03 05:55:17,421 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:17,421 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c534d6cb3070e98a48ea840ac1a7b2fb, regionState=CLOSED 2023-04-03 05:55:17,421 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501317421"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501317421"}]},"ts":"1680501317421"} 2023-04-03 05:55:17,423 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-03 05:55:17,423 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 in 159 msec 2023-04-03 05:55:17,423 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=c534d6cb3070e98a48ea840ac1a7b2fb, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,37397,1680501214665; forceNewPlan=false, retain=true 2023-04-03 05:55:17,574 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c534d6cb3070e98a48ea840ac1a7b2fb, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,574 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501317574"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501317574"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501317574"}]},"ts":"1680501317574"} 2023-04-03 05:55:17,575 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:55:17,731 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:17,731 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => c534d6cb3070e98a48ea840ac1a7b2fb, NAME => 'TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.', STARTKEY => '', ENDKEY => ''} 2023-04-03 05:55:17,731 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:17,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:55:17,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:17,732 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:17,733 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:17,733 DEBUG [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A 2023-04-03 05:55:17,733 DEBUG [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A 2023-04-03 05:55:17,734 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:55:17,734 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c534d6cb3070e98a48ea840ac1a7b2fb columnFamilyName A 2023-04-03 05:55:17,735 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.HStore(310): Store=c534d6cb3070e98a48ea840ac1a7b2fb/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:55:17,736 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:17,736 DEBUG [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B 2023-04-03 05:55:17,736 DEBUG [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B 2023-04-03 05:55:17,736 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:55:17,737 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c534d6cb3070e98a48ea840ac1a7b2fb columnFamilyName B 2023-04-03 05:55:17,737 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.HStore(310): Store=c534d6cb3070e98a48ea840ac1a7b2fb/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:55:17,737 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:17,738 DEBUG [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C 2023-04-03 05:55:17,738 DEBUG [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C 2023-04-03 05:55:17,738 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:55:17,738 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region c534d6cb3070e98a48ea840ac1a7b2fb columnFamilyName C 2023-04-03 05:55:17,738 INFO [StoreOpener-c534d6cb3070e98a48ea840ac1a7b2fb-1] regionserver.HStore(310): Store=c534d6cb3070e98a48ea840ac1a7b2fb/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:55:17,738 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:17,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:17,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:17,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 05:55:17,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:17,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened c534d6cb3070e98a48ea840ac1a7b2fb; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74824446, jitterRate=0.11497113108634949}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 05:55:17,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:17,743 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., pid=42, masterSystemTime=1680501317727 2023-04-03 05:55:17,744 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:17,744 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:17,744 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=c534d6cb3070e98a48ea840ac1a7b2fb, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,745 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501317744"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680501317744"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680501317744"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680501317744"}]},"ts":"1680501317744"} 2023-04-03 05:55:17,746 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-03 05:55:17,746 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 in 170 msec 2023-04-03 05:55:17,747 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-03 05:55:17,747 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c534d6cb3070e98a48ea840ac1a7b2fb, REOPEN/MOVE in 485 msec 2023-04-03 05:55:17,749 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-03 05:55:17,749 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-04-03 05:55:17,750 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 905 msec 2023-04-03 05:55:17,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-03 05:55:17,752 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x1cee66b5 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:17,757 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d800352, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:17,757 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x4c1d19b3 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:17,760 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2329054b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:17,761 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x14e151fb to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:17,764 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30951255, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:17,764 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x455a88b1 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:17,767 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5baa6768, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:17,767 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x722b2407 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:17,771 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63a535f3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:17,771 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x2d4bfeca to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:17,774 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@678b4c3d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:17,775 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x313c7ad6 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:17,778 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@43736f95, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:17,778 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x20d28b68 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:17,783 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@72eaaf6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:17,783 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x750d2a7d to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:17,787 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4130667f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:17,788 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x27313b38 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:17,791 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b73a361, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:17,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:17,800 DEBUG [hconnection-0x6824a131-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:17,801 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36580, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:17,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:17,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:17,806 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:17,806 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:17,806 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:17,806 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:17,806 DEBUG [hconnection-0xccd7c9e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:17,806 DEBUG [hconnection-0x2444f95a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:17,807 DEBUG [hconnection-0x4dc878da-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:17,807 DEBUG [hconnection-0xf4fe5a6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:17,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:17,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:17,809 DEBUG [hconnection-0x6fecdbfe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:17,810 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36600, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:17,811 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:17,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:17,811 DEBUG [hconnection-0x5fb1f202-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:17,811 DEBUG [hconnection-0x62c2222c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:17,810 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,810 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36586, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:17,811 DEBUG [hconnection-0x56531a3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:17,811 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36616, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:17,809 DEBUG [hconnection-0x1bc7c820-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:17,811 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,814 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:17,811 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36626, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:17,817 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36636, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:17,817 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36642, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:17,817 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36646, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:17,817 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36654, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:17,817 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36644, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:17,819 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:17,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:17,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:17,825 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:17,825 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:17,825 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:17,825 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:17,825 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:17,825 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:17,826 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:17,826 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:17,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:17,830 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:17,830 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 05:55:17,834 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:17,834 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:17,834 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:17,834 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:17,834 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:17,834 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:17,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:17,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:17,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501377845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501377846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,849 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:17,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:17,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501377846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501377847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,849 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:17,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501377848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:17,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501377950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:17,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:17,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501377950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501377950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,955 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:17,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501377950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:17,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:17,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501377951, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501378157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501378157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501378157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501378157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501378157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,247 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040302df828bfb56403ea0b9811252f48c57 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040302df828bfb56403ea0b9811252f48c57 2023-04-03 05:55:18,463 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501378461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501378461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501378463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501378463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501378464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,652 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/967a5854bc4847abaccbc56c1d0cc5f6 2023-04-03 05:55:18,689 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/1b6d8356fab140b09e2d7c8f187b1c87 2023-04-03 05:55:18,967 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501378965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,968 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501378966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501378967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501378967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:18,972 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:18,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501378969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,119 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/ae132765b13841d88a3a8c368fe954d6 2023-04-03 05:55:19,123 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/967a5854bc4847abaccbc56c1d0cc5f6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/967a5854bc4847abaccbc56c1d0cc5f6 2023-04-03 05:55:19,127 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/967a5854bc4847abaccbc56c1d0cc5f6, entries=150, sequenceid=16, filesize=25.1 K 2023-04-03 05:55:19,128 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/1b6d8356fab140b09e2d7c8f187b1c87 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/1b6d8356fab140b09e2d7c8f187b1c87 2023-04-03 05:55:19,132 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/1b6d8356fab140b09e2d7c8f187b1c87, entries=150, sequenceid=16, filesize=11.6 K 2023-04-03 05:55:19,132 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/ae132765b13841d88a3a8c368fe954d6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/ae132765b13841d88a3a8c368fe954d6 2023-04-03 05:55:19,137 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/ae132765b13841d88a3a8c368fe954d6, entries=150, sequenceid=16, filesize=11.6 K 2023-04-03 05:55:19,138 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for c534d6cb3070e98a48ea840ac1a7b2fb in 1309ms, sequenceid=16, compaction requested=false 2023-04-03 05:55:19,138 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:19,138 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:19,138 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:19,138 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:19,138 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,138 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:19,138 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:19,141 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,141 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:19,141 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:19,141 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:19,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:19,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:19,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ef4803e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:19,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:19,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:19,144 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:19,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:19,144 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:19,144 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:19,144 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:19,144 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,144 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:19,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:19,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:19,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:19,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:19,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:19,147 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:19,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:19,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:19,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:19,147 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:19,150 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:19,150 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:19,150 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:19,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:19,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:19,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:19,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:19,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:19,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:19,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:19,154 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:19,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:19,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:19,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:19,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:19,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:19,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:19,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:19,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:19,155 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:19,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:19,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:19,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:19,155 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:19,255 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:19,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:19,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:19,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:19,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:19,258 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:19,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:19,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:19,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:19,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:19,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:19,260 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:19,260 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:19,261 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,261 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:19,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:19,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:19,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:19,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:19,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:19,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:19,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:19,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:19,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:19,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:19,263 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:19,263 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:19,263 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 05:55:19,264 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:19,264 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:19,264 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:19,264 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:19,264 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:19,264 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:19,689 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304038450ef5478874bd5bfd841c4d85b549c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038450ef5478874bd5bfd841c4d85b549c 2023-04-03 05:55:19,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:19,972 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:19,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:19,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501379981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:19,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501379982, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:19,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501379983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:19,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501379984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:19,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:19,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501379985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501380086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,089 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501380088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,093 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501380090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501380092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501380092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,099 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/6c766ba7cc154efb8636ea90b88b6f7d 2023-04-03 05:55:20,114 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/a6e36d7ab1304c6494c3c4ba0cdb8b3b 2023-04-03 05:55:20,131 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/b2596ba975a74fec8cdad1431a588271 2023-04-03 05:55:20,136 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/6c766ba7cc154efb8636ea90b88b6f7d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/6c766ba7cc154efb8636ea90b88b6f7d 2023-04-03 05:55:20,140 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/6c766ba7cc154efb8636ea90b88b6f7d, entries=150, sequenceid=36, filesize=25.1 K 2023-04-03 05:55:20,141 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/a6e36d7ab1304c6494c3c4ba0cdb8b3b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/a6e36d7ab1304c6494c3c4ba0cdb8b3b 2023-04-03 05:55:20,150 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/a6e36d7ab1304c6494c3c4ba0cdb8b3b, entries=150, sequenceid=36, filesize=11.6 K 2023-04-03 05:55:20,150 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/b2596ba975a74fec8cdad1431a588271 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/b2596ba975a74fec8cdad1431a588271 2023-04-03 05:55:20,155 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/b2596ba975a74fec8cdad1431a588271, entries=150, sequenceid=36, filesize=11.6 K 2023-04-03 05:55:20,156 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c534d6cb3070e98a48ea840ac1a7b2fb in 893ms, sequenceid=36, compaction requested=false 2023-04-03 05:55:20,156 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:20,156 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:20,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:20,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:20,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:20,158 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,158 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,161 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:20,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@100d6f35[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:20,163 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:20,163 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,166 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,166 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:20,166 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:20,166 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:20,167 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,167 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:20,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:20,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:20,171 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:20,171 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:20,171 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:20,171 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:20,171 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:20,172 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,172 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,173 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,173 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,174 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,176 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:20,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:20,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,177 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:20,177 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:20,177 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,177 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,277 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:20,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:20,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:20,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:20,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:20,282 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:20,282 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:20,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:20,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:20,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,288 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,289 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:20,292 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,293 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:20,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:20,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:20,293 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:20,293 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:20,293 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:20,293 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:20,293 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:20,293 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:20,293 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:20,293 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 05:55:20,294 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:20,294 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,294 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:20,294 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:20,294 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,294 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:20,294 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:20,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:20,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:20,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:20,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:20,295 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:20,295 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:20,296 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,296 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,297 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,297 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:20,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:20,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49210026[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:20,301 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:20,302 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,307 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,307 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,307 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:20,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:20,308 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:20,309 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:20,309 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,309 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:20,310 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:20,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:20,312 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:20,312 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:20,312 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:20,312 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:20,312 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:20,318 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,318 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,318 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,319 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,321 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,324 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403f578b1143b48475893ba5440c1382b9d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f578b1143b48475893ba5440c1382b9d 2023-04-03 05:55:20,326 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,326 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,326 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,326 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,326 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,326 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:20,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:20,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,326 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,327 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:20,327 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:20,326 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,327 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:20,327 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,333 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/5aeddc9e9701408fb6cb13c2adccec7c 2023-04-03 05:55:20,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501380352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501380352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501380353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501380353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501380360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,427 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:20,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:20,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:20,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:20,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:20,430 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:20,430 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:20,430 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:20,430 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:20,431 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,431 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,432 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,432 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,432 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,432 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:20,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:20,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,433 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:20,433 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:20,433 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:20,433 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:20,433 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:20,433 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:20,433 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:20,433 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:20,433 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,434 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:20,434 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:20,434 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,434 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:20,434 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:20,434 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,434 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:20,434 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:20,441 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,441 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,442 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,442 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:20,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@499181db[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:20,443 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:20,443 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,444 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:20,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:20,444 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,445 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:20,446 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:20,446 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,446 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:20,447 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:20,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:20,449 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:20,449 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:20,449 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:20,449 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:20,449 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:20,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,453 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:20,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:20,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,462 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:20,463 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:20,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501380461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501380461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501380461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,464 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501380461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,469 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,469 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501380466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,563 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:20,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:20,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:20,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:20,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:20,566 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:20,566 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:20,566 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:20,566 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:20,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,568 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,568 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,568 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,568 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:20,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:20,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:20,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:20,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:20,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:20,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:20,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:20,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:20,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:20,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:20,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:20,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:20,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:20,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:20,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:20,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,572 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:20,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@133e0da7[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:20,574 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:20,574 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,575 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,575 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:20,575 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:20,575 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:20,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:20,578 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,578 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:20,578 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,578 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:20,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:20,581 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:20,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:20,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:20,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:20,581 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:20,583 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,583 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,584 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,585 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:20,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:20,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,588 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:20,588 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:20,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501380664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501380665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501380665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,667 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501380665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,672 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501380670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,688 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:20,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:20,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:20,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:20,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:20,692 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:20,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:20,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:20,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:20,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,693 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:20,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:20,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:20,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:20,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:20,695 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,695 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:20,695 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:20,695 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:20,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:20,698 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,698 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,699 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,699 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:20,700 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68b1bc32[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:20,701 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:20,701 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,702 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:20,702 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:20,702 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,702 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,703 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:20,704 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:20,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:20,707 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:20,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:20,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:20,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:20,707 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:20,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,709 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,710 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,713 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,713 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,713 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,713 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:20,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:20,713 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,713 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:20,714 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,714 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,714 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,714 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,714 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:20,714 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:20,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,746 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/6585be51f0b148359f80eb6662480240 2023-04-03 05:55:20,771 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 05:55:20,814 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:20,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:20,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:20,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:20,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:20,817 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:20,817 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:20,817 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:20,817 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:20,817 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,818 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,819 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,819 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,819 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,819 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:20,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,820 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:20,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:20,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:20,821 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:20,821 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:20,821 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:20,821 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:20,821 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:20,821 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:20,821 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,821 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:20,821 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:20,822 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,822 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:20,822 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:20,822 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,822 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:20,822 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:20,823 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,823 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:20,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:20,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2950b132[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:20,826 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:20,826 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,827 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:20,828 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,828 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:20,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:20,828 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:20,836 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,837 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,837 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:20,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:20,839 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:20,840 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:20,840 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:20,840 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:20,840 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:20,841 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,841 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,841 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,842 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,843 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:20,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:20,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,846 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,846 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:20,846 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:20,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:20,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,946 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:20,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:20,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:20,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:20,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:20,950 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:20,950 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:20,950 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:20,950 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:20,950 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,950 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,951 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,952 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:20,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,953 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,953 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:20,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:20,953 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:20,953 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:20,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:20,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:20,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:20,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:20,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:20,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:20,954 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:20,954 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:20,954 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:20,956 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,956 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:20,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c0a729a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:20,958 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:20,958 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,959 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:20,959 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:20,959 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,960 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:20,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:20,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,961 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:20,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:20,963 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:20,963 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:20,963 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:20,963 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:20,963 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:20,966 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,966 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:20,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,966 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:20,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:20,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:20,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:20,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,971 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,971 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,971 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:20,971 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:20,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:20,971 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:20,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:20,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:20,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:20,971 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,971 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:20,972 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:20,972 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:20,972 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:20,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:20,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:20,972 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:20,972 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:20,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501380968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501380969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501380969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501380969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:20,978 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:20,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501380973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,072 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:21,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:21,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:21,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:21,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:21,076 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:21,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:21,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:21,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:21,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:21,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:21,077 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:21,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:21,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:21,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,078 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:21,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:21,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:21,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:21,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:21,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:21,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:21,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:21,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:21,085 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:21,085 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:21,085 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:21,085 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:21,085 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:21,086 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,086 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:21,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:21,087 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:21,087 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:21,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:21,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:21,088 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fba0989[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:21,088 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:21,088 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:21,089 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:21,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:21,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:21,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:21,090 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:21,090 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,090 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:21,094 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:21,094 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,094 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:21,095 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:21,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:21,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:21,097 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:21,097 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:21,097 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:21,097 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:21,097 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:21,097 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:21,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:21,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:21,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:21,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:21,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:21,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:21,099 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:21,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:21,100 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:21,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:21,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:21,104 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:21,104 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:21,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:21,104 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:21,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:21,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:21,104 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,104 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:21,104 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:21,104 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:21,104 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:21,104 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:21,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:21,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:21,154 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/251d79074dd1496cb52410170137a8f1 2023-04-03 05:55:21,158 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/5aeddc9e9701408fb6cb13c2adccec7c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/5aeddc9e9701408fb6cb13c2adccec7c 2023-04-03 05:55:21,163 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/5aeddc9e9701408fb6cb13c2adccec7c, entries=250, sequenceid=50, filesize=38.7 K 2023-04-03 05:55:21,163 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/6585be51f0b148359f80eb6662480240 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/6585be51f0b148359f80eb6662480240 2023-04-03 05:55:21,167 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/6585be51f0b148359f80eb6662480240, entries=150, sequenceid=50, filesize=11.6 K 2023-04-03 05:55:21,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/251d79074dd1496cb52410170137a8f1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/251d79074dd1496cb52410170137a8f1 2023-04-03 05:55:21,173 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/251d79074dd1496cb52410170137a8f1, entries=150, sequenceid=50, filesize=11.6 K 2023-04-03 05:55:21,173 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c534d6cb3070e98a48ea840ac1a7b2fb in 880ms, sequenceid=50, compaction requested=true 2023-04-03 05:55:21,173 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:21,173 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:21,173 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:21,174 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:21,173 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:21,174 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:21,174 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 91122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:21,174 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:21,175 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/A is initiating minor compaction (all files) 2023-04-03 05:55:21,175 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/B is initiating minor compaction (all files) 2023-04-03 05:55:21,175 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/A in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:21,175 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/B in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:21,175 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/967a5854bc4847abaccbc56c1d0cc5f6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/6c766ba7cc154efb8636ea90b88b6f7d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/5aeddc9e9701408fb6cb13c2adccec7c] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=89.0 K 2023-04-03 05:55:21,175 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/1b6d8356fab140b09e2d7c8f187b1c87, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/a6e36d7ab1304c6494c3c4ba0cdb8b3b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/6585be51f0b148359f80eb6662480240] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=34.8 K 2023-04-03 05:55:21,175 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 967a5854bc4847abaccbc56c1d0cc5f6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680501317825 2023-04-03 05:55:21,176 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1b6d8356fab140b09e2d7c8f187b1c87, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680501317825 2023-04-03 05:55:21,177 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 6c766ba7cc154efb8636ea90b88b6f7d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680501317844 2023-04-03 05:55:21,177 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a6e36d7ab1304c6494c3c4ba0cdb8b3b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680501317844 2023-04-03 05:55:21,177 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 5aeddc9e9701408fb6cb13c2adccec7c, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680501319980 2023-04-03 05:55:21,177 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6585be51f0b148359f80eb6662480240, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680501319982 2023-04-03 05:55:21,183 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#B#compaction#361 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:21,187 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#A#compaction#362 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:21,205 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:21,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:21,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:21,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:21,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:21,209 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:21,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:21,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:21,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:21,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:21,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:21,211 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/ec7667cfee9b4cbf916e3d1a9d387e6d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/ec7667cfee9b4cbf916e3d1a9d387e6d 2023-04-03 05:55:21,213 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,213 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:21,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:21,214 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,214 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:21,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:21,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:21,216 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:21,216 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:21,216 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:21,216 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/B of c534d6cb3070e98a48ea840ac1a7b2fb into ec7667cfee9b4cbf916e3d1a9d387e6d(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:21,216 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:21,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:21,216 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/B, priority=13, startTime=1680501321173; duration=0sec 2023-04-03 05:55:21,217 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:21,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:21,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:21,217 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:21,217 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:21,217 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:21,217 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 05:55:21,217 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:21,217 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:21,217 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:21,217 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:21,217 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:21,217 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:21,218 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/C is initiating minor compaction (all files) 2023-04-03 05:55:21,218 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:21,218 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/C in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:21,218 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/ae132765b13841d88a3a8c368fe954d6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/b2596ba975a74fec8cdad1431a588271, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/251d79074dd1496cb52410170137a8f1] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=34.8 K 2023-04-03 05:55:21,219 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ae132765b13841d88a3a8c368fe954d6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680501317825 2023-04-03 05:55:21,219 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b2596ba975a74fec8cdad1431a588271, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1680501317844 2023-04-03 05:55:21,221 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 251d79074dd1496cb52410170137a8f1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680501319982 2023-04-03 05:55:21,229 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#C#compaction#363 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:21,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:21,479 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:21,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501381490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501381490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501381492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501381496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,506 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501381500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,605 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501381601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501381601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501381604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501381604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501381607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,615 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/a060c8bcb9e54308a0843a33f5cd5927 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/a060c8bcb9e54308a0843a33f5cd5927 2023-04-03 05:55:21,620 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/A of c534d6cb3070e98a48ea840ac1a7b2fb into a060c8bcb9e54308a0843a33f5cd5927(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:21,620 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:21,620 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/A, priority=13, startTime=1680501321173; duration=0sec 2023-04-03 05:55:21,620 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:21,649 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304032498124a1b444e4894a755f92f0b1aa5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032498124a1b444e4894a755f92f0b1aa5 2023-04-03 05:55:21,649 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/94924aa71f294f4a8ad06a143fd8d961 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/94924aa71f294f4a8ad06a143fd8d961 2023-04-03 05:55:21,653 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/C of c534d6cb3070e98a48ea840ac1a7b2fb into 94924aa71f294f4a8ad06a143fd8d961(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:21,653 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:21,653 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/C, priority=13, startTime=1680501321174; duration=0sec 2023-04-03 05:55:21,654 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:21,655 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/f0600a03c6904ebb9beceec1ad2676c7 2023-04-03 05:55:21,809 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501381806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,810 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501381807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501381810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501381810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:21,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:21,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501381814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,065 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/03d4ab4b4be145709a17dca7c757cd39 2023-04-03 05:55:22,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501382112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,115 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501382112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501382115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,119 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501382115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501382118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,474 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/0492a97cc929439481ab7466662dbb4b 2023-04-03 05:55:22,478 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/f0600a03c6904ebb9beceec1ad2676c7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/f0600a03c6904ebb9beceec1ad2676c7 2023-04-03 05:55:22,482 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/f0600a03c6904ebb9beceec1ad2676c7, entries=150, sequenceid=69, filesize=25.1 K 2023-04-03 05:55:22,482 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/03d4ab4b4be145709a17dca7c757cd39 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/03d4ab4b4be145709a17dca7c757cd39 2023-04-03 05:55:22,485 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/03d4ab4b4be145709a17dca7c757cd39, entries=150, sequenceid=69, filesize=11.6 K 2023-04-03 05:55:22,486 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/0492a97cc929439481ab7466662dbb4b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/0492a97cc929439481ab7466662dbb4b 2023-04-03 05:55:22,489 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/0492a97cc929439481ab7466662dbb4b, entries=150, sequenceid=69, filesize=11.6 K 2023-04-03 05:55:22,490 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c534d6cb3070e98a48ea840ac1a7b2fb in 1273ms, sequenceid=69, compaction requested=false 2023-04-03 05:55:22,490 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:22,490 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:22,490 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:22,492 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:22,492 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:22,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:22,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:22,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8f7460[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:22,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:22,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:22,495 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:22,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:22,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:22,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:22,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:22,496 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:22,496 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,496 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:22,497 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:22,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:22,497 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:22,497 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:22,497 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:22,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:22,501 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:22,501 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:22,501 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:22,501 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:22,501 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:22,501 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:22,504 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:22,504 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:22,504 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:22,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:22,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:22,505 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,507 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,510 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,510 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:22,510 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:22,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:22,510 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:22,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:22,510 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:22,510 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:22,511 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:22,510 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:22,511 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:22,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:22,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:22,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:22,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:22,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:22,611 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:22,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:22,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:22,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:22,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:22,614 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:22,614 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:22,614 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:22,614 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:22,614 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:22,614 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:22,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:22,615 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:22,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:22,616 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,616 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:22,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:22,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:22,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:22,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:22,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:22,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:22,617 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:22,617 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:22,617 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:22,617 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:22,617 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:22,617 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:22,617 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:22,617 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:55:22,617 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:22,617 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:22,618 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:22,618 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:22,618 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:22,618 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:22,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:22,623 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:22,664 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501382635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501382640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501382664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501382664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501382664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501382765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,767 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501382765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501382775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501382775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501382776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501382967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,971 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501382968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,983 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501382979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501382979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:22,984 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:22,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501382980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:23,029 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403e1d497cf0cfb4af18898763ab8ba4a96 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e1d497cf0cfb4af18898763ab8ba4a96 2023-04-03 05:55:23,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:23,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:23,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501383274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:23,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501383274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:23,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:23,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501383285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:23,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:23,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501383287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:23,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:23,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501383288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:23,435 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/c5179f03016f4a909e5f8ab6fcba33ff 2023-04-03 05:55:23,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:23,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501383783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:23,789 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:23,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501383785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:23,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:23,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501383793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:23,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:23,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501383794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:23,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:23,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501383795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:23,846 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/23f5cf55fbd54b22add9995952b7aae9 2023-04-03 05:55:24,256 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/aed2c33352984474a76143476b179865 2023-04-03 05:55:24,260 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/c5179f03016f4a909e5f8ab6fcba33ff as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/c5179f03016f4a909e5f8ab6fcba33ff 2023-04-03 05:55:24,263 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/c5179f03016f4a909e5f8ab6fcba33ff, entries=150, sequenceid=85, filesize=25.1 K 2023-04-03 05:55:24,267 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/23f5cf55fbd54b22add9995952b7aae9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/23f5cf55fbd54b22add9995952b7aae9 2023-04-03 05:55:24,274 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/23f5cf55fbd54b22add9995952b7aae9, entries=150, sequenceid=85, filesize=11.6 K 2023-04-03 05:55:24,275 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/aed2c33352984474a76143476b179865 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/aed2c33352984474a76143476b179865 2023-04-03 05:55:24,278 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/aed2c33352984474a76143476b179865, entries=150, sequenceid=85, filesize=11.6 K 2023-04-03 05:55:24,279 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c534d6cb3070e98a48ea840ac1a7b2fb in 1662ms, sequenceid=85, compaction requested=true 2023-04-03 05:55:24,279 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:24,279 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:24,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:24,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,281 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:24,281 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:24,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:24,281 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:24,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:24,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:24,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46fa8122[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:24,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:24,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:24,284 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:24,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:24,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:24,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:24,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:24,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:24,285 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,285 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:24,286 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:24,286 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:24,286 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:24,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:24,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:24,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:24,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:24,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:24,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:24,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:24,293 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:24,293 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:24,293 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:24,293 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:24,293 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:24,296 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:24,296 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:24,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:24,296 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:24,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:24,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:24,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:24,298 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:24,299 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:24,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,302 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:24,302 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:24,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:24,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:24,302 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:24,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:24,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:24,303 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:24,303 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:24,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:24,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:24,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:24,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:24,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:24,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:24,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:24,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:24,403 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:24,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:24,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:24,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:24,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:24,407 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:24,407 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:24,407 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:24,407 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:24,407 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:24,407 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:24,408 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,409 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,409 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:24,409 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:24,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:24,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:24,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:24,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:24,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:24,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:24,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:24,410 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:24,411 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:24,411 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:24,411 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:24,411 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:24,411 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 05:55:24,411 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:24,411 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:24,411 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:24,411 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:24,412 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:24,412 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:24,433 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403b128963db0d749748c83d9814d30ee38 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b128963db0d749748c83d9814d30ee38 2023-04-03 05:55:24,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:24,795 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:24,817 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:24,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501384812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:24,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501384813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:24,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501384813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,820 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:24,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501384814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:24,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501384817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,837 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=103, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/047abde445704acb999017011ac39ad7 2023-04-03 05:55:24,849 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/7bd8273ea6fe42bf870b3a5d10ab4971 2023-04-03 05:55:24,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:24,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501384918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:24,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501384919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:24,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501384919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:24,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501384921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:24,930 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:24,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501384925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501385122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501385124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501385124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501385125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,135 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501385132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,258 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=103 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/a6622053a456403b90736776f23945ab 2023-04-03 05:55:25,262 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/047abde445704acb999017011ac39ad7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/047abde445704acb999017011ac39ad7 2023-04-03 05:55:25,265 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/047abde445704acb999017011ac39ad7, entries=150, sequenceid=103, filesize=25.1 K 2023-04-03 05:55:25,266 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/7bd8273ea6fe42bf870b3a5d10ab4971 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/7bd8273ea6fe42bf870b3a5d10ab4971 2023-04-03 05:55:25,269 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/7bd8273ea6fe42bf870b3a5d10ab4971, entries=150, sequenceid=103, filesize=11.6 K 2023-04-03 05:55:25,270 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/a6622053a456403b90736776f23945ab as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/a6622053a456403b90736776f23945ab 2023-04-03 05:55:25,273 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/a6622053a456403b90736776f23945ab, entries=150, sequenceid=103, filesize=11.6 K 2023-04-03 05:55:25,274 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c534d6cb3070e98a48ea840ac1a7b2fb in 863ms, sequenceid=103, compaction requested=true 2023-04-03 05:55:25,274 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:25,274 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:25,275 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,275 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:25,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:25,276 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:25,276 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:25,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:25,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79dba5c5[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:25,277 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:25,277 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:25,279 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:25,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:25,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:25,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:25,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:25,279 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,280 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:25,281 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:25,281 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:25,281 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:25,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:25,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:25,283 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:25,283 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:25,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:25,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:25,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:25,283 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:25,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:25,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:25,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:25,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:25,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:25,285 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:25,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:25,286 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:25,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:25,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:25,289 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:25,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:25,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:25,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:25,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:25,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:25,289 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:25,289 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:25,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:25,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:25,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:25,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:25,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:25,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:25,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:25,389 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:25,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:25,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:25,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:25,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:25,392 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:25,393 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:25,393 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:25,393 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:25,393 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:25,393 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:25,395 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,395 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:25,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:25,395 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,396 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:25,396 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:25,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:25,396 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:25,396 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:25,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:25,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:25,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:25,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:25,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:25,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:25,397 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:25,397 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:25,397 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:55:25,397 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:25,397 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:25,398 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:25,398 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:25,398 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:25,398 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:25,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:25,433 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:25,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501385451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,465 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501385458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501385458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501385459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501385459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,563 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501385559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,570 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501385565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501385566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501385566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,571 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501385567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,770 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501385765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,773 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501385772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501385773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,774 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501385773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:25,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501385773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:25,809 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403d955865e1ec34a8c9b0af855658a66e8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d955865e1ec34a8c9b0af855658a66e8 2023-04-03 05:55:26,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:26,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501386072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:26,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:26,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501386075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:26,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:26,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501386076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:26,080 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:26,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501386077, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:26,087 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:26,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501386081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:26,214 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=117, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/4715b049af6b47c4b5a25c58a41ca605 2023-04-03 05:55:26,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:26,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501386582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:26,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:26,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501386583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:26,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:26,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501386584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:26,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501386584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:26,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:26,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501386589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:26,628 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/cc2ed3d9b0874bf0be770c73f8d17dbd 2023-04-03 05:55:27,036 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/ad4b7bc9aa044fd6abfc1ba5059b5926 2023-04-03 05:55:27,040 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/4715b049af6b47c4b5a25c58a41ca605 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/4715b049af6b47c4b5a25c58a41ca605 2023-04-03 05:55:27,044 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/4715b049af6b47c4b5a25c58a41ca605, entries=150, sequenceid=117, filesize=25.1 K 2023-04-03 05:55:27,044 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/cc2ed3d9b0874bf0be770c73f8d17dbd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/cc2ed3d9b0874bf0be770c73f8d17dbd 2023-04-03 05:55:27,049 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/cc2ed3d9b0874bf0be770c73f8d17dbd, entries=150, sequenceid=117, filesize=11.6 K 2023-04-03 05:55:27,050 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/ad4b7bc9aa044fd6abfc1ba5059b5926 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/ad4b7bc9aa044fd6abfc1ba5059b5926 2023-04-03 05:55:27,053 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/ad4b7bc9aa044fd6abfc1ba5059b5926, entries=150, sequenceid=117, filesize=11.6 K 2023-04-03 05:55:27,053 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c534d6cb3070e98a48ea840ac1a7b2fb in 1656ms, sequenceid=117, compaction requested=true 2023-04-03 05:55:27,054 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:27,054 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:27,054 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:27,054 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:27,054 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,054 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:27,054 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:27,057 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,057 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:27,058 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:27,058 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:27,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:27,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26f21d36[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:27,059 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:27,059 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:27,060 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:27,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:27,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:27,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:27,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:27,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:27,061 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,061 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:27,064 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,064 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:27,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:27,064 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:27,064 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:27,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:27,066 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:27,066 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:27,066 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:27,066 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:27,067 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:27,068 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:27,068 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:27,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:27,068 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:27,069 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:27,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:27,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:27,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:27,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:27,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:27,072 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:27,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:27,072 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:27,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:27,072 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:27,072 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:27,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:27,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:27,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:27,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:27,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:27,172 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:27,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:27,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:27,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:27,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:27,176 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:27,176 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:27,176 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:27,176 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:27,176 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:27,176 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:27,177 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,179 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,179 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:27,179 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:27,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:27,180 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:27,180 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:27,180 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:27,180 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:27,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:27,181 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:27,181 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:27,184 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:27,184 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:27,184 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:27,184 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 05:55:27,184 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:27,184 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:27,184 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:27,184 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:27,185 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:27,185 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:27,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:27,588 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:27,608 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403a768688a637043f895963b326e84115e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a768688a637043f895963b326e84115e 2023-04-03 05:55:27,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501387603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501387607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501387608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501387609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501387610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501387712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501387713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,721 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501387713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,721 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501387718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,722 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501387718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501387922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,930 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501387923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501387923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,930 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501387923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:27,931 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:27,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501387924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,014 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=135, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/3886c838a2014d6c82c919069ebdf222 2023-04-03 05:55:28,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501388229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,235 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501388231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501388231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501388233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501388233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,428 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/5a789fd751994018873fec1e603c6369 2023-04-03 05:55:28,439 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/b0bfb23e31b44e34a405ceb8b6305ed7 2023-04-03 05:55:28,442 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/3886c838a2014d6c82c919069ebdf222 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/3886c838a2014d6c82c919069ebdf222 2023-04-03 05:55:28,446 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/3886c838a2014d6c82c919069ebdf222, entries=150, sequenceid=135, filesize=25.2 K 2023-04-03 05:55:28,447 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/5a789fd751994018873fec1e603c6369 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/5a789fd751994018873fec1e603c6369 2023-04-03 05:55:28,450 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/5a789fd751994018873fec1e603c6369, entries=150, sequenceid=135, filesize=11.7 K 2023-04-03 05:55:28,451 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/b0bfb23e31b44e34a405ceb8b6305ed7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/b0bfb23e31b44e34a405ceb8b6305ed7 2023-04-03 05:55:28,453 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/b0bfb23e31b44e34a405ceb8b6305ed7, entries=150, sequenceid=135, filesize=11.7 K 2023-04-03 05:55:28,454 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c534d6cb3070e98a48ea840ac1a7b2fb in 1270ms, sequenceid=135, compaction requested=true 2023-04-03 05:55:28,454 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:28,454 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:28,454 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:28,454 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:28,454 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,455 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:28,455 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:28,458 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:28,458 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:28,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:28,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37cf51fc[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:28,459 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:28,459 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:28,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:28,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:28,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:28,461 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:28,461 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:28,461 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:28,461 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,461 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:28,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:28,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:28,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:28,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:28,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:28,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:28,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:28,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:28,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:28,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:28,465 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:28,465 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:28,465 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:28,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:28,466 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:28,467 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:28,467 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:28,467 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:28,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:28,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:28,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:28,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,469 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:28,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:28,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:28,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:28,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:28,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:28,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:28,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:28,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:28,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:28,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:28,473 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:28,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:28,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:28,473 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:28,473 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:28,573 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:28,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:28,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:28,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:28,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:28,577 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:28,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:28,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:28,577 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:28,578 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:28,578 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:28,579 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:28,579 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:28,579 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,579 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:28,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:28,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:28,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:28,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:28,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:28,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:28,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:28,580 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:28,580 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:28,580 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:28,580 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:28,580 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:28,580 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:28,580 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:28,581 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:55:28,581 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:28,581 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:28,581 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:28,581 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:28,581 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:28,581 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:28,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:28,741 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:28,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501388759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501388768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501388769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501388771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,788 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501388773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501388874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501388880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501388880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501388881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:28,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501388889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:28,992 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403f4795502cbe746dca64035f18ff7a396 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f4795502cbe746dca64035f18ff7a396 2023-04-03 05:55:28,998 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=149, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/ded3e993b67e4c84aed13e2988314baa 2023-04-03 05:55:29,019 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/28ca6359289a41aca9b9b00f2996fd7e 2023-04-03 05:55:29,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501389085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501389085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501389089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,096 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501389090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,098 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501389095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501389391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501389392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501389398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501389398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,407 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501389400, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,438 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=149 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/439f3a674c774b3b9dfaedb521c7ec06 2023-04-03 05:55:29,442 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/ded3e993b67e4c84aed13e2988314baa as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/ded3e993b67e4c84aed13e2988314baa 2023-04-03 05:55:29,446 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/ded3e993b67e4c84aed13e2988314baa, entries=150, sequenceid=149, filesize=25.3 K 2023-04-03 05:55:29,446 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/28ca6359289a41aca9b9b00f2996fd7e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/28ca6359289a41aca9b9b00f2996fd7e 2023-04-03 05:55:29,457 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/28ca6359289a41aca9b9b00f2996fd7e, entries=150, sequenceid=149, filesize=11.8 K 2023-04-03 05:55:29,458 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/439f3a674c774b3b9dfaedb521c7ec06 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/439f3a674c774b3b9dfaedb521c7ec06 2023-04-03 05:55:29,462 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/439f3a674c774b3b9dfaedb521c7ec06, entries=150, sequenceid=149, filesize=11.8 K 2023-04-03 05:55:29,463 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c534d6cb3070e98a48ea840ac1a7b2fb in 882ms, sequenceid=149, compaction requested=true 2023-04-03 05:55:29,463 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:29,463 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:29,463 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:29,464 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:29,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:29,473 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:29,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:29,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:29,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:29,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:29,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@252e25b6[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:29,475 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:29,475 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:29,476 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:29,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:29,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:29,476 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,477 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:29,477 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:29,477 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:29,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:29,477 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:29,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:29,483 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:29,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:29,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:29,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:29,483 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:29,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:29,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:29,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:29,486 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:29,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:29,486 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:29,487 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:29,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:29,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:29,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:29,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:29,489 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:29,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:29,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:29,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,490 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:29,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:29,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:29,490 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:29,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:29,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:29,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:29,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:29,590 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:29,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:29,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:29,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:29,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:29,593 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:29,593 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:29,593 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:29,593 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:29,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:29,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:29,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:29,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:29,595 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:29,595 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,595 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:29,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:29,596 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:29,596 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:29,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:29,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:29,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:29,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:29,597 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:29,597 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:29,597 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 05:55:29,597 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:29,597 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:29,597 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:29,597 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:29,597 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:29,597 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:29,609 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403c5376c82c6674267b13abb533d2e2270 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c5376c82c6674267b13abb533d2e2270 2023-04-03 05:55:29,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:29,907 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:29,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501389923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501389924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,939 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501389932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501389933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:29,951 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:29,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501389937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,014 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=167, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/bb3cbe3e39ae4b499f1782f26f0ae6e1 2023-04-03 05:55:30,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501390038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501390040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501390040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501390043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501390052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501390245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501390249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501390250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501390253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,267 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501390261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,424 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/4e969c98cbc8446ca5fb728512426eb2 2023-04-03 05:55:30,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501390553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501390553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501390558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501390561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:30,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501390569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,835 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=167 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/232cae5ea21f47ecb0314196b1c84183 2023-04-03 05:55:30,838 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/bb3cbe3e39ae4b499f1782f26f0ae6e1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/bb3cbe3e39ae4b499f1782f26f0ae6e1 2023-04-03 05:55:30,842 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/bb3cbe3e39ae4b499f1782f26f0ae6e1, entries=150, sequenceid=167, filesize=25.3 K 2023-04-03 05:55:30,842 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/4e969c98cbc8446ca5fb728512426eb2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/4e969c98cbc8446ca5fb728512426eb2 2023-04-03 05:55:30,846 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/4e969c98cbc8446ca5fb728512426eb2, entries=150, sequenceid=167, filesize=11.8 K 2023-04-03 05:55:30,846 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/232cae5ea21f47ecb0314196b1c84183 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/232cae5ea21f47ecb0314196b1c84183 2023-04-03 05:55:30,849 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/232cae5ea21f47ecb0314196b1c84183, entries=150, sequenceid=167, filesize=11.8 K 2023-04-03 05:55:30,850 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c534d6cb3070e98a48ea840ac1a7b2fb in 1253ms, sequenceid=167, compaction requested=true 2023-04-03 05:55:30,850 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:30,850 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:30,850 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:30,850 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:30,850 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,850 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:30,850 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:30,852 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:30,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:30,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:30,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:30,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f495b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:30,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:30,854 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:30,856 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:30,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:30,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:30,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:30,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:30,856 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,856 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:30,856 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:30,859 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:30,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:30,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:30,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:30,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:30,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:30,862 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:30,862 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:30,862 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:30,862 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:30,862 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:30,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:30,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:30,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:30,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:30,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:30,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:30,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:30,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:30,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:30,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:30,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,866 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:30,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,872 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,872 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:30,872 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:30,872 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:30,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:30,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:30,872 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:30,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:30,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:30,872 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:30,872 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,872 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:30,872 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:30,872 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:30,872 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:30,872 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:30,873 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:30,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:30,973 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:30,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:30,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:30,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:30,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:30,976 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:30,976 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:30,976 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:30,976 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:30,976 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:30,976 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:30,978 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,978 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:30,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:30,978 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:30,978 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:30,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:30,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:30,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:30,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:30,979 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:30,979 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:30,979 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:30,979 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:30,979 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:30,979 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:30,979 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:30,979 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:55:30,980 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:30,980 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:30,980 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:30,980 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:30,980 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:30,980 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:30,995 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304037c201ae8e34f46c5983eadaa1e548440 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037c201ae8e34f46c5983eadaa1e548440 2023-04-03 05:55:31,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:31,067 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:31,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501391090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,108 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501391100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501391107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501391107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501391108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501391209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501391209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501391220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,231 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501391226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501391226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,400 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=181, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/e396ea453aad4b9caebcfa787613110e 2023-04-03 05:55:31,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501391416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501391417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501391426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501391432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,440 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501391434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501391722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,730 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501391723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501391735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,748 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501391741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:31,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501391743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:31,809 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/db652ec5ba4d49be824a45d314f1c44a 2023-04-03 05:55:32,218 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/2e5ebf006e194fe88952e327ef465f51 2023-04-03 05:55:32,222 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/e396ea453aad4b9caebcfa787613110e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/e396ea453aad4b9caebcfa787613110e 2023-04-03 05:55:32,225 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/e396ea453aad4b9caebcfa787613110e, entries=150, sequenceid=181, filesize=25.3 K 2023-04-03 05:55:32,226 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/db652ec5ba4d49be824a45d314f1c44a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/db652ec5ba4d49be824a45d314f1c44a 2023-04-03 05:55:32,229 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/db652ec5ba4d49be824a45d314f1c44a, entries=150, sequenceid=181, filesize=11.8 K 2023-04-03 05:55:32,230 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/2e5ebf006e194fe88952e327ef465f51 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/2e5ebf006e194fe88952e327ef465f51 2023-04-03 05:55:32,232 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/2e5ebf006e194fe88952e327ef465f51, entries=150, sequenceid=181, filesize=11.8 K 2023-04-03 05:55:32,233 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for c534d6cb3070e98a48ea840ac1a7b2fb in 1254ms, sequenceid=181, compaction requested=true 2023-04-03 05:55:32,233 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:32,233 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:32,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:32,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:32,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:32,235 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:32,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:32,236 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 05:55:32,236 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:32,236 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:32,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:32,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:32,237 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:32,237 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:32,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3752ffef[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:32,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:32,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:32,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:32,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:32,240 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,244 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,244 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:32,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:32,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:32,247 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:32,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:32,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:32,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:32,247 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:32,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,250 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:32,252 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:32,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:32,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:32,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:32,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,253 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,253 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:32,253 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,253 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:32,253 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,253 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501392271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501392276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,292 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501392277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,296 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501392281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,297 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501392282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,353 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:32,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:32,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:32,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:32,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:32,356 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:32,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:32,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:32,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:32,357 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,357 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,358 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:32,358 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:32,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:32,358 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,358 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:32,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,358 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:32,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:32,359 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:32,359 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:32,359 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:32,359 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:32,359 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:32,359 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:32,359 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,359 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:32,359 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:32,359 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,359 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:32,360 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:32,360 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,360 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:32,360 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:32,365 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,365 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,365 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,365 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:32,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7adf0848[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:32,366 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:32,366 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,367 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,367 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:32,367 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:32,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,368 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:32,368 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:32,369 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:32,369 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,369 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,369 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:32,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:32,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:32,371 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:32,371 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:32,371 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:32,371 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:32,371 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:32,373 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,373 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,373 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,373 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,374 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,377 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:32,377 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,377 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:32,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:32,377 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:32,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:32,377 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:32,377 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:32,377 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:32,377 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,377 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,377 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,377 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,377 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,391 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501392383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36636 deadline: 1680501392393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36654 deadline: 1680501392393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501392397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36600 deadline: 1680501392397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,477 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:32,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:32,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:32,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:32,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:32,481 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:32,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:32,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:32,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:32,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,482 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:32,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:32,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:32,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:32,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:32,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:32,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:32,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:32,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:32,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:32,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:32,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:32,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:32,484 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,484 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:32,484 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:32,484 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:32,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:32,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:32,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:32,486 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:32,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e088130[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:32,487 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:32,487 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,489 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:32,489 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:32,489 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:32,489 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,489 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:32,491 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,491 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,491 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,491 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:32,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:32,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:32,494 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:32,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:32,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:32,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:32,494 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:32,498 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,498 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,498 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,499 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,500 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,502 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,502 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,502 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:32,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:32,502 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:32,502 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:32,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:32,502 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,503 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:32,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:32,503 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,502 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,503 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,503 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:32,503 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,602 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36580 deadline: 1680501392594, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,603 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:32,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:32,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:32,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:32,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:32,606 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:32,606 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:32,606 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:32,606 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:32,607 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,607 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,609 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:32,609 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:32,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:32,609 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,609 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:32,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:32,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,610 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:32,611 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:32,611 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:32,611 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:32,611 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:32,612 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:32,612 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:32,612 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:32,612 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,612 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:32,612 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:32,612 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,612 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:32,612 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:32,612 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,612 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:32,612 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:32,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501392603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,613 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,614 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,614 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,614 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:32,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c6203b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:32,615 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:32,615 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,616 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:32,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:32,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:32,616 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,616 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:32,617 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:32,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,617 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501392604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36616 deadline: 1680501392606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501392607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:32,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:32,620 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:32,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:32,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:32,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:32,620 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:32,622 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,622 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,622 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,623 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,624 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,626 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:32,626 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:32,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:32,626 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:32,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:32,626 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,626 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,626 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:32,627 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:32,626 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,627 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,627 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,627 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,649 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304032ea6ac149c0b4f61857ca87a87d42a38 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032ea6ac149c0b4f61857ca87a87d42a38 2023-04-03 05:55:32,727 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:32,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:32,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:32,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:32,730 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:32,730 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:32,730 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:32,731 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:32,732 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,732 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,733 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,733 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:32,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:32,733 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,733 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:32,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:32,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:32,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:32,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:32,734 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:32,734 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:32,734 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:32,735 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,735 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:32,735 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:32,735 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:32,735 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:32,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,738 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:32,738 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:32,738 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75cb10eb[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:32,738 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:32,738 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,740 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:32,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:32,740 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:32,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:32,742 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:32,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:32,744 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:32,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:32,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:32,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:32,744 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:32,746 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,746 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,746 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,748 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,751 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,751 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,751 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,751 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,751 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:32,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:32,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:32,751 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:32,751 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,751 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:32,751 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:32,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:32,752 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:32,751 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,752 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,852 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:32,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:32,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:32,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:32,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:32,855 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:32,855 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:32,855 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:32,855 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:32,856 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,856 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,857 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,857 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:32,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:32,858 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,858 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:32,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:32,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:32,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:32,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:32,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:32,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:32,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:32,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:32,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:32,859 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,860 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:32,860 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:32,860 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:32,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:32,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:32,860 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:32,861 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,861 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,861 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,861 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:32,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:32,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7754cb52[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:32,863 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:32,863 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:32,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:32,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:32,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:32,865 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:32,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:32,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:32,868 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:32,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:32,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:32,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:32,868 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:32,870 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,870 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,870 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,872 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,872 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,873 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,877 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,877 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,877 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,877 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:32,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:32,877 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:32,877 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:32,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:32,877 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:32,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:32,877 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,877 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:32,877 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,877 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,878 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,913 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501392905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,924 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501392916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501392919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501392919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:32,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501392921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,978 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:32,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:32,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:32,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:32,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:32,980 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:32,980 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:32,981 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:32,981 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:32,981 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,981 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,982 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,982 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:32,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:32,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:32,982 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,982 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:32,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:32,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:32,984 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:32,984 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,984 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:32,984 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:32,984 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:32,984 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:32,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,986 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,987 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,987 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:32,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d3fc589[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:32,988 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:32,988 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,989 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:32,990 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:32,990 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:32,990 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:32,990 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,990 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:32,991 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:32,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:32,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:32,993 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:32,993 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:32,993 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:32,993 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:32,993 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:32,995 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,995 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,995 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:32,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:32,995 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,997 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:32,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:32,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,000 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:33,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:33,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:33,001 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:33,001 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:33,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,055 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=201, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/026a35d267634eb39eac51192cce899a 2023-04-03 05:55:33,101 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:33,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:33,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:33,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:33,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:33,104 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:33,104 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,104 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:33,104 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:33,105 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,105 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,106 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,106 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,106 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,106 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:33,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:33,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,107 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:33,107 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:33,107 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:33,107 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,107 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:33,107 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:33,107 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:33,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:33,109 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,109 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,109 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,109 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:33,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:33,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791cf9ed[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:33,111 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:33,111 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,112 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,112 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:33,112 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:33,112 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:33,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,112 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:33,115 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:33,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:33,117 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:33,117 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:33,117 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:33,117 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:33,117 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:33,118 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,118 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,118 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,119 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,120 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,122 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,122 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,122 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,122 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,122 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:33,122 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:33,122 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,122 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,122 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,122 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:33,122 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:33,123 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,123 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,223 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:33,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:33,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:33,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:33,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:33,226 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:33,226 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,226 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:33,226 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:33,227 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,227 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,228 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,228 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,228 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,228 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:33,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,229 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,229 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:33,229 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:33,229 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:33,230 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,230 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:33,230 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:33,230 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:33,230 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:33,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:33,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547de26d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:33,234 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:33,234 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,235 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,235 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:33,236 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:33,236 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:33,236 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,236 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:33,237 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,237 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,237 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,237 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:33,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:33,239 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:33,239 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:33,239 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:33,239 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:33,239 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:33,241 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,241 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,241 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,241 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,242 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,248 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:33,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:33,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:33,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,248 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:33,249 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:33,349 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:33,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:33,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:33,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:33,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:33,352 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:33,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:33,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:33,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,353 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,353 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,355 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:33,355 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:33,355 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:33,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:33,355 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,355 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:33,355 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:33,355 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:33,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,357 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:33,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a2adeb[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:33,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:33,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,361 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:33,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:33,361 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:33,361 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:33,364 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,364 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:33,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:33,367 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:33,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:33,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:33,367 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:33,367 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:33,368 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,368 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,368 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,369 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,370 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:33,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:33,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:33,372 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:33,372 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:33,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:33,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501393416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,431 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:33,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501393425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:33,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501393426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,436 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:33,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501393428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:33,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501393430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,463 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/96a8b5f0f0774c179493a35d354e7cd9 2023-04-03 05:55:33,473 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:33,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:33,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:33,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:33,478 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:33,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:33,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:33,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,482 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:33,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:33,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:33,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:33,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:33,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:33,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:33,488 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,488 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:33,488 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:33,488 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:33,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:33,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,490 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,491 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,491 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:33,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6652335b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:33,492 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:33,492 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,493 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,493 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:33,493 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,493 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:33,493 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:33,493 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:33,495 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:33,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:33,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:33,497 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:33,497 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:33,497 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:33,497 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:33,497 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:33,509 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,509 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,509 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,510 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,515 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,515 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,515 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,515 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,515 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,515 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:33,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:33,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,515 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,515 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,515 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,516 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:33,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,516 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:33,516 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:33,616 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:33,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:33,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:33,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:33,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:33,619 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:33,619 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,619 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:33,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:33,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,621 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,621 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,622 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,622 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:33,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:33,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:33,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:33,623 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,623 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:33,623 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:33,623 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:33,623 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,624 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:33,624 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:33,625 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,625 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,625 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,625 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:33,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:33,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@789f6ded[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:33,627 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:33,627 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,628 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:33,628 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:33,629 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,629 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:33,631 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:33,631 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,631 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,631 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:33,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:33,633 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:33,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:33,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:33,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:33,633 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:33,634 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,634 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,634 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,636 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:33,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:33,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,639 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,639 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:33,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,640 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,639 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:33,640 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,740 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:33,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:33,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:33,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:33,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:33,743 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:33,743 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,743 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:33,743 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:33,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,745 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,745 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,745 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,745 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:33,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:33,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:33,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:33,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:33,746 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:33,746 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,746 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:33,746 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:33,746 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:33,746 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,747 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:33,747 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:33,747 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,747 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:33,747 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:33,747 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,747 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:33,747 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:33,752 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,752 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,753 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,753 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:33,754 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43ca9619[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:33,754 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:33,754 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,755 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,755 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,756 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:33,756 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:33,756 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:33,756 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:33,757 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:33,757 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,757 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,757 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:33,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:33,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:33,759 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:33,759 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:33,759 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:33,759 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:33,760 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:33,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,761 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,762 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:33,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:33,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,766 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,766 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:33,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,766 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:33,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,766 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,866 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:33,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:33,868 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:33,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:33,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:33,869 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:33,869 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,869 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:33,869 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:33,870 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,870 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,871 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,871 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,871 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,871 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:33,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,872 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:33,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,872 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:33,872 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:33,872 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:33,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:33,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:33,873 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:33,873 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:33,873 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,873 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:33,873 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:33,873 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,873 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:33,873 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:33,873 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,873 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:33,873 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:33,873 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=201 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/d68610381db5481294907073e211b9dd 2023-04-03 05:55:33,874 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,874 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,875 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,875 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:33,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,877 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/026a35d267634eb39eac51192cce899a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/026a35d267634eb39eac51192cce899a 2023-04-03 05:55:33,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:33,877 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:33,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5317b67e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:33,877 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:33,879 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,879 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:33,879 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:33,879 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:33,880 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/026a35d267634eb39eac51192cce899a, entries=200, sequenceid=201, filesize=32.1 K 2023-04-03 05:55:33,881 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/96a8b5f0f0774c179493a35d354e7cd9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/96a8b5f0f0774c179493a35d354e7cd9 2023-04-03 05:55:33,882 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:33,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:33,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,884 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/96a8b5f0f0774c179493a35d354e7cd9, entries=150, sequenceid=201, filesize=11.8 K 2023-04-03 05:55:33,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:33,885 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:33,885 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:33,885 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/d68610381db5481294907073e211b9dd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/d68610381db5481294907073e211b9dd 2023-04-03 05:55:33,885 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:33,885 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:33,885 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:33,885 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:33,887 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,887 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,887 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:33,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:33,888 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:33,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/d68610381db5481294907073e211b9dd, entries=150, sequenceid=201, filesize=11.8 K 2023-04-03 05:55:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,889 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for c534d6cb3070e98a48ea840ac1a7b2fb in 1653ms, sequenceid=201, compaction requested=true 2023-04-03 05:55:33,889 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,889 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:33,890 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-03 05:55:33,889 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:33,890 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:33,890 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-03 05:55:33,890 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:33,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:33,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,893 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 119663 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-03 05:55:33,893 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/B is initiating minor compaction (all files) 2023-04-03 05:55:33,893 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/B in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,893 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/ec7667cfee9b4cbf916e3d1a9d387e6d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/03d4ab4b4be145709a17dca7c757cd39, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/23f5cf55fbd54b22add9995952b7aae9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/7bd8273ea6fe42bf870b3a5d10ab4971, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/cc2ed3d9b0874bf0be770c73f8d17dbd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/5a789fd751994018873fec1e603c6369, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/28ca6359289a41aca9b9b00f2996fd7e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/4e969c98cbc8446ca5fb728512426eb2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/db652ec5ba4d49be824a45d314f1c44a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/96a8b5f0f0774c179493a35d354e7cd9] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=116.9 K 2023-04-03 05:55:33,894 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 265253 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-03 05:55:33,894 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/A is initiating minor compaction (all files) 2023-04-03 05:55:33,894 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:33,894 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/A in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,894 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:33,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:33,894 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/a060c8bcb9e54308a0843a33f5cd5927, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/f0600a03c6904ebb9beceec1ad2676c7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/c5179f03016f4a909e5f8ab6fcba33ff, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/047abde445704acb999017011ac39ad7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/4715b049af6b47c4b5a25c58a41ca605, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/3886c838a2014d6c82c919069ebdf222, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/ded3e993b67e4c84aed13e2988314baa, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/bb3cbe3e39ae4b499f1782f26f0ae6e1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/e396ea453aad4b9caebcfa787613110e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/026a35d267634eb39eac51192cce899a] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=259.0 K 2023-04-03 05:55:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:33,894 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ec7667cfee9b4cbf916e3d1a9d387e6d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680501319982 2023-04-03 05:55:33,894 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting a060c8bcb9e54308a0843a33f5cd5927, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680501319982 2023-04-03 05:55:33,895 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 03d4ab4b4be145709a17dca7c757cd39, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680501320352 2023-04-03 05:55:33,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:33,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:33,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:33,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:33,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:33,895 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:33,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:33,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:33,895 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting f0600a03c6904ebb9beceec1ad2676c7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680501320352 2023-04-03 05:55:33,896 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 23f5cf55fbd54b22add9995952b7aae9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680501321489 2023-04-03 05:55:33,896 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:33,896 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:33,896 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting c5179f03016f4a909e5f8ab6fcba33ff, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680501321489 2023-04-03 05:55:33,896 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 7bd8273ea6fe42bf870b3a5d10ab4971, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680501322630 2023-04-03 05:55:33,896 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 047abde445704acb999017011ac39ad7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680501322630 2023-04-03 05:55:33,897 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting cc2ed3d9b0874bf0be770c73f8d17dbd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680501324802 2023-04-03 05:55:33,897 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 4715b049af6b47c4b5a25c58a41ca605, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680501324802 2023-04-03 05:55:33,897 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 5a789fd751994018873fec1e603c6369, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1680501325443 2023-04-03 05:55:33,897 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 3886c838a2014d6c82c919069ebdf222, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1680501325443 2023-04-03 05:55:33,897 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 28ca6359289a41aca9b9b00f2996fd7e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1680501327592 2023-04-03 05:55:33,898 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting ded3e993b67e4c84aed13e2988314baa, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1680501327592 2023-04-03 05:55:33,898 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 4e969c98cbc8446ca5fb728512426eb2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680501328768 2023-04-03 05:55:33,898 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting bb3cbe3e39ae4b499f1782f26f0ae6e1, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680501328768 2023-04-03 05:55:33,898 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting db652ec5ba4d49be824a45d314f1c44a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680501329923 2023-04-03 05:55:33,898 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e396ea453aad4b9caebcfa787613110e, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680501329923 2023-04-03 05:55:33,898 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 96a8b5f0f0774c179493a35d354e7cd9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680501331106 2023-04-03 05:55:33,898 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 026a35d267634eb39eac51192cce899a, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680501331106 2023-04-03 05:55:33,912 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#B#compaction#392 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:33,915 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#A#compaction#391 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:33,925 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/3d29762dae974b90888dbafab0332e17 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/3d29762dae974b90888dbafab0332e17 2023-04-03 05:55:33,929 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/B of c534d6cb3070e98a48ea840ac1a7b2fb into 3d29762dae974b90888dbafab0332e17(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:33,929 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:33,929 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/B, priority=6, startTime=1680501333890; duration=0sec 2023-04-03 05:55:33,929 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:33,929 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-03 05:55:33,941 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 119663 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-03 05:55:33,942 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/C is initiating minor compaction (all files) 2023-04-03 05:55:33,942 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/C in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:33,942 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/94924aa71f294f4a8ad06a143fd8d961, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/0492a97cc929439481ab7466662dbb4b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/aed2c33352984474a76143476b179865, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/a6622053a456403b90736776f23945ab, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/ad4b7bc9aa044fd6abfc1ba5059b5926, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/b0bfb23e31b44e34a405ceb8b6305ed7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/439f3a674c774b3b9dfaedb521c7ec06, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/232cae5ea21f47ecb0314196b1c84183, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/2e5ebf006e194fe88952e327ef465f51, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/d68610381db5481294907073e211b9dd] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=116.9 K 2023-04-03 05:55:33,942 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 94924aa71f294f4a8ad06a143fd8d961, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680501319982 2023-04-03 05:55:33,943 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0492a97cc929439481ab7466662dbb4b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1680501320352 2023-04-03 05:55:33,944 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting aed2c33352984474a76143476b179865, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1680501321489 2023-04-03 05:55:33,944 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a6622053a456403b90736776f23945ab, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=103, earliestPutTs=1680501322630 2023-04-03 05:55:33,944 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ad4b7bc9aa044fd6abfc1ba5059b5926, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680501324802 2023-04-03 05:55:33,944 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b0bfb23e31b44e34a405ceb8b6305ed7, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1680501325443 2023-04-03 05:55:33,945 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 439f3a674c774b3b9dfaedb521c7ec06, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=149, earliestPutTs=1680501327592 2023-04-03 05:55:33,945 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 232cae5ea21f47ecb0314196b1c84183, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=167, earliestPutTs=1680501328768 2023-04-03 05:55:33,945 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 2e5ebf006e194fe88952e327ef465f51, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680501329923 2023-04-03 05:55:33,945 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting d68610381db5481294907073e211b9dd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680501331106 2023-04-03 05:55:33,955 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#C#compaction#393 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:33,996 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:33,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:33,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:33,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:33,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:33,999 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:33,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:33,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:33,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:34,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:34,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:34,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:34,002 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,002 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:34,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:34,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:34,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:34,003 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:34,003 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:34,003 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:34,004 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:34,004 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:34,004 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:34,004 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:34,004 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 05:55:34,004 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:34,004 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:34,004 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:34,004 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:34,004 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:34,004 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:34,016 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040364ead20bbc204c7f827b997b32b0b60e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040364ead20bbc204c7f827b997b32b0b60e 2023-04-03 05:55:34,022 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=214, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/5733e9c02f5448c092cb91c78ee42729 2023-04-03 05:55:34,363 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/4ac619d8a53341188e98e52814365d90 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4ac619d8a53341188e98e52814365d90 2023-04-03 05:55:34,368 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/C of c534d6cb3070e98a48ea840ac1a7b2fb into 4ac619d8a53341188e98e52814365d90(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:34,368 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:34,368 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/C, priority=6, startTime=1680501333890; duration=0sec 2023-04-03 05:55:34,368 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:34,438 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/b670961ea3f44ab9a943041092aa0c2e 2023-04-03 05:55:34,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:34,441 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:34,446 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=214 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/52c7c74cf13041188b3454a382efe314 2023-04-03 05:55:34,450 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/5733e9c02f5448c092cb91c78ee42729 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/5733e9c02f5448c092cb91c78ee42729 2023-04-03 05:55:34,453 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/5733e9c02f5448c092cb91c78ee42729, entries=150, sequenceid=214, filesize=25.3 K 2023-04-03 05:55:34,454 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/b670961ea3f44ab9a943041092aa0c2e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/b670961ea3f44ab9a943041092aa0c2e 2023-04-03 05:55:34,457 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/b670961ea3f44ab9a943041092aa0c2e, entries=150, sequenceid=214, filesize=11.8 K 2023-04-03 05:55:34,458 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/52c7c74cf13041188b3454a382efe314 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/52c7c74cf13041188b3454a382efe314 2023-04-03 05:55:34,461 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/52c7c74cf13041188b3454a382efe314, entries=150, sequenceid=214, filesize=11.8 K 2023-04-03 05:55:34,461 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=80.51 KB/82440 for c534d6cb3070e98a48ea840ac1a7b2fb in 457ms, sequenceid=214, compaction requested=false 2023-04-03 05:55:34,461 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:34,462 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:34,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:34,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:34,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:34,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:34,464 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,464 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:34,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:34,465 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,465 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 05:55:34,465 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:34,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:34,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:34,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24a88fac[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:34,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,468 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:34,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:34,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:34,473 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:34,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:34,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,474 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:34,475 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,475 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:34,475 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:34,475 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:34,475 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:34,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:34,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:34,477 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:34,477 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:34,477 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:34,477 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:34,477 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:34,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:34,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:34,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:34,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:34,480 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,488 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,488 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:34,488 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:34,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:34,488 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:34,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:34,488 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:34,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:34,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:34,488 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,488 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,488 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:34,488 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:34,488 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:34,488 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501394489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501394490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,506 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501394498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501394498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501394499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,589 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:34,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:34,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:34,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:34,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:34,592 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:34,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:34,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:34,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:34,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,593 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:34,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:34,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:34,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:34,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:34,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:34,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:34,595 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:34,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:34,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:34,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:34,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:34,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:34,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:34,596 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:34,597 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:34,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:34,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,597 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:34,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:34,599 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:34,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@275b6ed1[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:34,599 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,600 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,600 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,601 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:34,600 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:34,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:34,605 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:34,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:34,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:34,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:34,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:34,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501394600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:34,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:34,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501394605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:34,609 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:34,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:34,609 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:34,609 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:34,609 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:34,609 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:34,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:34,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:34,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,610 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:34,611 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:34,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:34,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,614 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,614 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,614 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:34,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:34,614 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:34,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:34,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:34,614 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:34,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:34,614 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:34,614 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,614 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,614 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,614 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,614 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:34,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:34,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:34,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501394607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,615 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:34,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501394608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501394608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,715 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:34,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:34,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:34,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:34,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:34,719 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:34,719 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:34,719 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:34,719 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:34,720 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,720 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,721 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,721 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:34,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:34,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:34,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:34,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:34,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,725 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:34,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:34,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:34,732 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:34,732 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:34,732 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:34,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:34,740 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:34,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:34,740 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:34,740 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:34,740 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:34,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:34,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:34,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:34,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:34,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,743 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:34,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:34,744 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:34,744 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:34,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:34,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30b65e7d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:34,746 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:34,746 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,747 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,748 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:34,748 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:34,748 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:34,748 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,748 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:34,749 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:34,749 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:34,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:34,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:34,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,753 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/ff74b293d69b4a5a83bb0f4ec8ac6a0e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/ff74b293d69b4a5a83bb0f4ec8ac6a0e 2023-04-03 05:55:34,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:34,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:34,755 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:34,755 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:34,755 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:34,755 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:34,755 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:34,758 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,758 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:34,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,759 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:34,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:34,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:34,760 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/A of c534d6cb3070e98a48ea840ac1a7b2fb into ff74b293d69b4a5a83bb0f4ec8ac6a0e(size=25.7 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:34,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,760 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:34,760 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/A, priority=6, startTime=1680501333889; duration=0sec 2023-04-03 05:55:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:34,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,760 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,761 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:34,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:34,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:34,771 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:34,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:34,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:34,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:34,771 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:34,771 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:34,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:34,815 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501394807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501394808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,821 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501394816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501394816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:34,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501394817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,871 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:34,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:34,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:34,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:34,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:34,874 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:34,874 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:34,874 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:34,874 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:34,875 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,875 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,876 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:34,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:34,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:34,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:34,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:34,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:34,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:34,877 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:34,877 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:34,877 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:34,877 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:34,877 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:34,877 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:34,877 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:34,878 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:34,878 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:34,878 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:34,879 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:34,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:34,879 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:34,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:34,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:34,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e7cba53[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:34,881 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:34,881 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,882 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:34,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:34,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:34,882 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,882 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:34,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,885 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:34,885 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:34,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:34,885 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:34,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:34,887 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403e3c4345e783e428ead4b0c5e714128ec to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e3c4345e783e428ead4b0c5e714128ec 2023-04-03 05:55:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:34,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:34,888 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:34,888 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:34,888 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:34,888 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:34,889 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:34,889 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:34,890 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,890 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:34,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,890 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:34,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:34,890 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:34,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:34,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:34,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,892 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=232, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/b7b1a8c4ccbc4dc2a9d096134c8992c4 2023-04-03 05:55:34,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:34,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:34,895 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:34,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:34,895 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:34,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:34,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:34,896 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:34,896 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:34,896 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:34,896 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:34,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:34,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:34,896 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:34,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,900 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/5850122705b74f9f91d0f87afb8c109f 2023-04-03 05:55:34,996 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:34,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:34,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:34,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:34,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:34,999 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:34,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:34,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:34,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:34,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:34,999 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:35,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:35,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:35,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:35,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:35,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:35,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:35,002 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:35,002 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:35,002 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:35,002 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:35,002 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:35,002 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:35,003 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:35,003 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:35,003 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:35,003 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:35,003 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:35,003 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,003 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:35,003 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:35,005 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:35,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:35,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:35,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:35,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:35,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:35,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:35,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e498bdf[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:35,007 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:35,007 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,008 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,009 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:35,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:35,009 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:35,009 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:35,010 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:35,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:35,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:35,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:35,012 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:35,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:35,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:35,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:35,012 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:35,017 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,017 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:35,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:35,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,018 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:35,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,019 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:35,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:35,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:35,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:35,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:35,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:35,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:35,021 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:35,021 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:35,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:35,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:35,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:35,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501395117, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,121 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:35,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:35,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:35,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:35,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:35,125 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:35,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:35,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:35,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:35,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,126 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,126 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:35,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:35,126 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,126 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:35,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501395122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:35,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:35,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:35,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:35,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:35,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:35,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:35,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501395124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,129 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:35,129 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:35,129 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:35,129 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:35,129 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:35,129 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:35,129 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:35,129 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:35,129 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,129 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:35,129 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:35,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,132 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:35,132 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,132 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:35,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:35,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501395129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42185721[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:35,134 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:35,134 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,134 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501395130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,135 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:35,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:35,136 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:35,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:35,136 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:35,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:35,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:35,137 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:35,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:35,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:35,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:35,137 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:35,138 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,138 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,138 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:35,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:35,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,139 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:35,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,140 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:35,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:35,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:35,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:35,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:35,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:35,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:35,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:35,143 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,143 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:35,144 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,144 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:35,144 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:35,244 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:35,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:35,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:35,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:35,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:35,258 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:35,258 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:35,258 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:35,258 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:35,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,261 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:35,261 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:35,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:35,261 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,261 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:35,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:35,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:35,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:35,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:35,263 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:35,263 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:35,263 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:35,263 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:35,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:35,265 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,265 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:35,265 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:35,265 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:35,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:35,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40dfbca9[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:35,267 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:35,267 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,268 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:35,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:35,273 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:35,276 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:35,276 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:35,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:35,278 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:35,278 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:35,278 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:35,278 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:35,278 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:35,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:35,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:35,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:35,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:35,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:35,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:35,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:35,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:35,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,286 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:35,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:35,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:35,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:35,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:35,286 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:35,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:35,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:35,286 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:35,309 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/2c294ee7055048a7b61bf3f199fd31c1 2023-04-03 05:55:35,313 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/b7b1a8c4ccbc4dc2a9d096134c8992c4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/b7b1a8c4ccbc4dc2a9d096134c8992c4 2023-04-03 05:55:35,318 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/b7b1a8c4ccbc4dc2a9d096134c8992c4, entries=200, sequenceid=232, filesize=32.1 K 2023-04-03 05:55:35,319 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/5850122705b74f9f91d0f87afb8c109f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/5850122705b74f9f91d0f87afb8c109f 2023-04-03 05:55:35,329 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/5850122705b74f9f91d0f87afb8c109f, entries=150, sequenceid=232, filesize=11.8 K 2023-04-03 05:55:35,330 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/2c294ee7055048a7b61bf3f199fd31c1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/2c294ee7055048a7b61bf3f199fd31c1 2023-04-03 05:55:35,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/2c294ee7055048a7b61bf3f199fd31c1, entries=150, sequenceid=232, filesize=11.8 K 2023-04-03 05:55:35,334 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for c534d6cb3070e98a48ea840ac1a7b2fb in 869ms, sequenceid=232, compaction requested=true 2023-04-03 05:55:35,334 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:35,334 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:35,334 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:35,334 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:35,334 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:35,334 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:35,335 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85122 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:35,335 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/A is initiating minor compaction (all files) 2023-04-03 05:55:35,335 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/A in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:35,335 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/ff74b293d69b4a5a83bb0f4ec8ac6a0e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/5733e9c02f5448c092cb91c78ee42729, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/b7b1a8c4ccbc4dc2a9d096134c8992c4] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=83.1 K 2023-04-03 05:55:35,336 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:35,336 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting ff74b293d69b4a5a83bb0f4ec8ac6a0e, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680501331106 2023-04-03 05:55:35,336 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/B is initiating minor compaction (all files) 2023-04-03 05:55:35,336 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/B in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:35,336 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/3d29762dae974b90888dbafab0332e17, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/b670961ea3f44ab9a943041092aa0c2e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/5850122705b74f9f91d0f87afb8c109f] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=35.7 K 2023-04-03 05:55:35,336 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 5733e9c02f5448c092cb91c78ee42729, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680501332275 2023-04-03 05:55:35,336 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3d29762dae974b90888dbafab0332e17, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680501331106 2023-04-03 05:55:35,336 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting b7b1a8c4ccbc4dc2a9d096134c8992c4, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680501334457 2023-04-03 05:55:35,336 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b670961ea3f44ab9a943041092aa0c2e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680501332275 2023-04-03 05:55:35,337 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 5850122705b74f9f91d0f87afb8c109f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680501334457 2023-04-03 05:55:35,353 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#B#compaction#401 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:35,366 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#A#compaction#400 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:35,387 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:35,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:35,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:35,408 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/e75065c5352d41069f228b3b8100e1c3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/e75065c5352d41069f228b3b8100e1c3 2023-04-03 05:55:35,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:35,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:35,412 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:35,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:35,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:35,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:35,413 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,413 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,414 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:35,414 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:35,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:35,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,415 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,415 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/B of c534d6cb3070e98a48ea840ac1a7b2fb into e75065c5352d41069f228b3b8100e1c3(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:35,415 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:35,415 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/B, priority=13, startTime=1680501335334; duration=0sec 2023-04-03 05:55:35,415 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:35,415 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:35,415 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:35,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:35,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:35,417 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:35,417 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/C is initiating minor compaction (all files) 2023-04-03 05:55:35,417 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/C in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:35,417 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4ac619d8a53341188e98e52814365d90, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/52c7c74cf13041188b3454a382efe314, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/2c294ee7055048a7b61bf3f199fd31c1] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=35.7 K 2023-04-03 05:55:35,418 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 4ac619d8a53341188e98e52814365d90, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=201, earliestPutTs=1680501331106 2023-04-03 05:55:35,418 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 52c7c74cf13041188b3454a382efe314, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=214, earliestPutTs=1680501332275 2023-04-03 05:55:35,419 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 2c294ee7055048a7b61bf3f199fd31c1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680501334457 2023-04-03 05:55:35,420 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:35,420 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:35,420 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:35,420 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:35,420 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:35,420 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:35,420 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:35,420 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:35,420 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:35,420 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 05:55:35,420 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:35,421 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:35,421 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:35,421 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:35,421 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:35,421 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:35,433 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#C#compaction#402 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:35,484 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040354a6a545b565468693eb6f4c003816a5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040354a6a545b565468693eb6f4c003816a5 2023-04-03 05:55:35,487 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/254e6015715d4b85b5da4d8ecee17bad as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/254e6015715d4b85b5da4d8ecee17bad 2023-04-03 05:55:35,491 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=249, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/e9a282a072ce4eef92c62d77418db755 2023-04-03 05:55:35,494 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/A of c534d6cb3070e98a48ea840ac1a7b2fb into 254e6015715d4b85b5da4d8ecee17bad(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:35,494 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:35,494 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/A, priority=13, startTime=1680501335334; duration=0sec 2023-04-03 05:55:35,494 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:35,500 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/64c30bf0392343f0981dfd13f74be232 2023-04-03 05:55:35,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:35,623 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:35,683 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501395670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501395671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,684 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501395672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501395682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501395684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501395785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501395785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501395785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:35,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501395791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501395791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,859 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/f8b71582314747d5bdca77d6b6194455 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/f8b71582314747d5bdca77d6b6194455 2023-04-03 05:55:35,863 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/C of c534d6cb3070e98a48ea840ac1a7b2fb into f8b71582314747d5bdca77d6b6194455(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:35,863 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:35,863 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/C, priority=13, startTime=1680501335334; duration=0sec 2023-04-03 05:55:35,864 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:35,898 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-03 05:55:35,913 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=249 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/1f8547f484104532a7db26e016876aab 2023-04-03 05:55:35,917 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/e9a282a072ce4eef92c62d77418db755 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/e9a282a072ce4eef92c62d77418db755 2023-04-03 05:55:35,950 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/967a5854bc4847abaccbc56c1d0cc5f6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/6c766ba7cc154efb8636ea90b88b6f7d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/5aeddc9e9701408fb6cb13c2adccec7c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/a060c8bcb9e54308a0843a33f5cd5927, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/f0600a03c6904ebb9beceec1ad2676c7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/c5179f03016f4a909e5f8ab6fcba33ff, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/047abde445704acb999017011ac39ad7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/4715b049af6b47c4b5a25c58a41ca605, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/3886c838a2014d6c82c919069ebdf222, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/ded3e993b67e4c84aed13e2988314baa, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/bb3cbe3e39ae4b499f1782f26f0ae6e1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/e396ea453aad4b9caebcfa787613110e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/026a35d267634eb39eac51192cce899a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/ff74b293d69b4a5a83bb0f4ec8ac6a0e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/5733e9c02f5448c092cb91c78ee42729, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/b7b1a8c4ccbc4dc2a9d096134c8992c4] to archive 2023-04-03 05:55:35,950 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/e9a282a072ce4eef92c62d77418db755, entries=150, sequenceid=249, filesize=25.3 K 2023-04-03 05:55:35,950 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/64c30bf0392343f0981dfd13f74be232 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/64c30bf0392343f0981dfd13f74be232 2023-04-03 05:55:35,950 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:55:35,952 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/967a5854bc4847abaccbc56c1d0cc5f6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/967a5854bc4847abaccbc56c1d0cc5f6 2023-04-03 05:55:35,953 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/6c766ba7cc154efb8636ea90b88b6f7d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/6c766ba7cc154efb8636ea90b88b6f7d 2023-04-03 05:55:35,955 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/64c30bf0392343f0981dfd13f74be232, entries=150, sequenceid=249, filesize=11.8 K 2023-04-03 05:55:35,956 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/1f8547f484104532a7db26e016876aab as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/1f8547f484104532a7db26e016876aab 2023-04-03 05:55:35,956 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/5aeddc9e9701408fb6cb13c2adccec7c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/5aeddc9e9701408fb6cb13c2adccec7c 2023-04-03 05:55:35,957 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/a060c8bcb9e54308a0843a33f5cd5927 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/a060c8bcb9e54308a0843a33f5cd5927 2023-04-03 05:55:35,958 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/f0600a03c6904ebb9beceec1ad2676c7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/f0600a03c6904ebb9beceec1ad2676c7 2023-04-03 05:55:35,959 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/1f8547f484104532a7db26e016876aab, entries=150, sequenceid=249, filesize=11.8 K 2023-04-03 05:55:35,959 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/c5179f03016f4a909e5f8ab6fcba33ff to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/c5179f03016f4a909e5f8ab6fcba33ff 2023-04-03 05:55:35,960 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for c534d6cb3070e98a48ea840ac1a7b2fb in 540ms, sequenceid=249, compaction requested=false 2023-04-03 05:55:35,961 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:35,961 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:35,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:35,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:35,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:35,961 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:35,962 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/047abde445704acb999017011ac39ad7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/047abde445704acb999017011ac39ad7 2023-04-03 05:55:35,964 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,964 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,964 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:35,964 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:35,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:35,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:35,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:35,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:35,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:35,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bca38c3[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:35,966 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:35,966 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,967 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,967 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:35,967 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:35,967 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:35,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:35,968 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,968 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:35,968 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:35,968 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:35,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:35,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:35,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:35,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:35,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:35,971 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:35,972 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:35,972 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:35,972 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:35,972 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:35,973 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,973 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:35,973 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/4715b049af6b47c4b5a25c58a41ca605 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/4715b049af6b47c4b5a25c58a41ca605 2023-04-03 05:55:35,973 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:35,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:35,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,974 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/3886c838a2014d6c82c919069ebdf222 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/3886c838a2014d6c82c919069ebdf222 2023-04-03 05:55:35,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:35,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:35,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:35,975 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,975 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/ded3e993b67e4c84aed13e2988314baa to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/ded3e993b67e4c84aed13e2988314baa 2023-04-03 05:55:35,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:35,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,977 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:35,978 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/bb3cbe3e39ae4b499f1782f26f0ae6e1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/bb3cbe3e39ae4b499f1782f26f0ae6e1 2023-04-03 05:55:35,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:35,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,979 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/e396ea453aad4b9caebcfa787613110e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/e396ea453aad4b9caebcfa787613110e 2023-04-03 05:55:35,982 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/026a35d267634eb39eac51192cce899a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/026a35d267634eb39eac51192cce899a 2023-04-03 05:55:35,983 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,983 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:35,983 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:35,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:35,983 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:35,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:35,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:35,983 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:35,983 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:35,983 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:35,983 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:35,983 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:35,983 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,983 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:35,983 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:35,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:35,984 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:35,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:35,985 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/ff74b293d69b4a5a83bb0f4ec8ac6a0e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/ff74b293d69b4a5a83bb0f4ec8ac6a0e 2023-04-03 05:55:35,988 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/5733e9c02f5448c092cb91c78ee42729 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/5733e9c02f5448c092cb91c78ee42729 2023-04-03 05:55:35,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:35,990 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-03 05:55:35,990 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/b7b1a8c4ccbc4dc2a9d096134c8992c4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/b7b1a8c4ccbc4dc2a9d096134c8992c4 2023-04-03 05:55:35,990 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:35,991 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:35,991 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:35,991 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:35,991 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:35,991 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:36,009 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/1b6d8356fab140b09e2d7c8f187b1c87, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/a6e36d7ab1304c6494c3c4ba0cdb8b3b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/ec7667cfee9b4cbf916e3d1a9d387e6d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/6585be51f0b148359f80eb6662480240, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/03d4ab4b4be145709a17dca7c757cd39, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/23f5cf55fbd54b22add9995952b7aae9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/7bd8273ea6fe42bf870b3a5d10ab4971, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/cc2ed3d9b0874bf0be770c73f8d17dbd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/5a789fd751994018873fec1e603c6369, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/28ca6359289a41aca9b9b00f2996fd7e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/4e969c98cbc8446ca5fb728512426eb2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/db652ec5ba4d49be824a45d314f1c44a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/3d29762dae974b90888dbafab0332e17, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/96a8b5f0f0774c179493a35d354e7cd9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/b670961ea3f44ab9a943041092aa0c2e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/5850122705b74f9f91d0f87afb8c109f] to archive 2023-04-03 05:55:36,011 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:55:36,021 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,021 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/1b6d8356fab140b09e2d7c8f187b1c87 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/1b6d8356fab140b09e2d7c8f187b1c87 2023-04-03 05:55:36,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501396014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501396016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501396019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501396021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,023 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/a6e36d7ab1304c6494c3c4ba0cdb8b3b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/a6e36d7ab1304c6494c3c4ba0cdb8b3b 2023-04-03 05:55:36,025 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/ec7667cfee9b4cbf916e3d1a9d387e6d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/ec7667cfee9b4cbf916e3d1a9d387e6d 2023-04-03 05:55:36,026 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/6585be51f0b148359f80eb6662480240 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/6585be51f0b148359f80eb6662480240 2023-04-03 05:55:36,027 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/03d4ab4b4be145709a17dca7c757cd39 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/03d4ab4b4be145709a17dca7c757cd39 2023-04-03 05:55:36,028 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/23f5cf55fbd54b22add9995952b7aae9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/23f5cf55fbd54b22add9995952b7aae9 2023-04-03 05:55:36,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501396021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,031 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/7bd8273ea6fe42bf870b3a5d10ab4971 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/7bd8273ea6fe42bf870b3a5d10ab4971 2023-04-03 05:55:36,032 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/cc2ed3d9b0874bf0be770c73f8d17dbd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/cc2ed3d9b0874bf0be770c73f8d17dbd 2023-04-03 05:55:36,032 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304031a5fd1e411b44a9495d9969b0da89332 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031a5fd1e411b44a9495d9969b0da89332 2023-04-03 05:55:36,034 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/5a789fd751994018873fec1e603c6369 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/5a789fd751994018873fec1e603c6369 2023-04-03 05:55:36,035 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/28ca6359289a41aca9b9b00f2996fd7e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/28ca6359289a41aca9b9b00f2996fd7e 2023-04-03 05:55:36,037 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/4e969c98cbc8446ca5fb728512426eb2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/4e969c98cbc8446ca5fb728512426eb2 2023-04-03 05:55:36,038 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/db652ec5ba4d49be824a45d314f1c44a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/db652ec5ba4d49be824a45d314f1c44a 2023-04-03 05:55:36,039 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/3d29762dae974b90888dbafab0332e17 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/3d29762dae974b90888dbafab0332e17 2023-04-03 05:55:36,040 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/96a8b5f0f0774c179493a35d354e7cd9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/96a8b5f0f0774c179493a35d354e7cd9 2023-04-03 05:55:36,041 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/b670961ea3f44ab9a943041092aa0c2e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/b670961ea3f44ab9a943041092aa0c2e 2023-04-03 05:55:36,042 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/5850122705b74f9f91d0f87afb8c109f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/5850122705b74f9f91d0f87afb8c109f 2023-04-03 05:55:36,046 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/ae132765b13841d88a3a8c368fe954d6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/b2596ba975a74fec8cdad1431a588271, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/94924aa71f294f4a8ad06a143fd8d961, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/251d79074dd1496cb52410170137a8f1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/0492a97cc929439481ab7466662dbb4b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/aed2c33352984474a76143476b179865, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/a6622053a456403b90736776f23945ab, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/ad4b7bc9aa044fd6abfc1ba5059b5926, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/b0bfb23e31b44e34a405ceb8b6305ed7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/439f3a674c774b3b9dfaedb521c7ec06, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/232cae5ea21f47ecb0314196b1c84183, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/2e5ebf006e194fe88952e327ef465f51, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4ac619d8a53341188e98e52814365d90, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/d68610381db5481294907073e211b9dd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/52c7c74cf13041188b3454a382efe314, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/2c294ee7055048a7b61bf3f199fd31c1] to archive 2023-04-03 05:55:36,057 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:55:36,058 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/ae132765b13841d88a3a8c368fe954d6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/ae132765b13841d88a3a8c368fe954d6 2023-04-03 05:55:36,059 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/b2596ba975a74fec8cdad1431a588271 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/b2596ba975a74fec8cdad1431a588271 2023-04-03 05:55:36,061 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/94924aa71f294f4a8ad06a143fd8d961 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/94924aa71f294f4a8ad06a143fd8d961 2023-04-03 05:55:36,062 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/251d79074dd1496cb52410170137a8f1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/251d79074dd1496cb52410170137a8f1 2023-04-03 05:55:36,062 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/0492a97cc929439481ab7466662dbb4b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/0492a97cc929439481ab7466662dbb4b 2023-04-03 05:55:36,063 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/aed2c33352984474a76143476b179865 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/aed2c33352984474a76143476b179865 2023-04-03 05:55:36,064 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/a6622053a456403b90736776f23945ab to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/a6622053a456403b90736776f23945ab 2023-04-03 05:55:36,065 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/ad4b7bc9aa044fd6abfc1ba5059b5926 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/ad4b7bc9aa044fd6abfc1ba5059b5926 2023-04-03 05:55:36,066 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/b0bfb23e31b44e34a405ceb8b6305ed7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/b0bfb23e31b44e34a405ceb8b6305ed7 2023-04-03 05:55:36,067 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/439f3a674c774b3b9dfaedb521c7ec06 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/439f3a674c774b3b9dfaedb521c7ec06 2023-04-03 05:55:36,068 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/232cae5ea21f47ecb0314196b1c84183 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/232cae5ea21f47ecb0314196b1c84183 2023-04-03 05:55:36,069 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/2e5ebf006e194fe88952e327ef465f51 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/2e5ebf006e194fe88952e327ef465f51 2023-04-03 05:55:36,070 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4ac619d8a53341188e98e52814365d90 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4ac619d8a53341188e98e52814365d90 2023-04-03 05:55:36,071 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/d68610381db5481294907073e211b9dd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/d68610381db5481294907073e211b9dd 2023-04-03 05:55:36,072 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/52c7c74cf13041188b3454a382efe314 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/52c7c74cf13041188b3454a382efe314 2023-04-03 05:55:36,073 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/2c294ee7055048a7b61bf3f199fd31c1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/2c294ee7055048a7b61bf3f199fd31c1 2023-04-03 05:55:36,083 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:36,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:36,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:36,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:36,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:36,086 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:36,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:36,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:36,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,087 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,087 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,088 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,088 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:36,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,088 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:36,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,088 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:36,089 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:36,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:36,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:36,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:36,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:36,089 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,089 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:36,089 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:36,089 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:36,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:36,090 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,090 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:36,090 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:36,091 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,091 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,091 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,091 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:36,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:36,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@92f4316[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:36,093 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:36,093 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,095 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,095 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:36,096 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:36,096 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:36,096 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,096 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:36,097 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:36,097 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,097 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,097 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:36,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:36,099 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:36,099 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:36,099 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:36,099 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:36,099 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:36,100 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,100 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,100 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,101 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,102 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:36,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:36,104 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,104 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:36,103 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,104 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:36,104 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,104 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,104 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:36,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,127 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501396122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501396123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501396123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501396123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501396131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,204 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:36,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:36,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:36,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:36,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:36,207 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:36,207 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,207 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:36,207 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:36,208 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,208 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,210 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,210 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,211 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,211 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,211 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:36,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,211 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:36,212 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:36,212 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:36,212 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,212 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:36,212 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:36,212 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:36,212 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:36,214 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,214 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,214 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,214 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:36,215 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@98ebc4d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:36,215 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:36,215 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,216 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:36,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:36,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:36,216 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,216 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:36,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:36,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,217 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:36,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:36,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:36,219 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:36,220 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:36,220 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:36,220 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:36,220 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:36,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,222 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,223 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,225 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,225 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,225 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,225 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,225 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,225 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:36,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:36,226 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,226 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,226 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,226 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,226 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:36,226 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:36,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,326 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:36,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:36,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:36,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:36,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:36,329 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:36,329 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,329 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:36,329 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:36,330 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,330 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,331 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,331 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,331 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:36,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:36,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,332 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:36,332 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,332 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:36,332 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:36,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501396329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,332 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:36,332 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,332 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:36,334 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:36,334 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:36,334 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,334 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:36,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501396329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,334 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:36,334 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,334 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:36,334 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:36,334 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501396329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,334 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:36,335 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:36,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501396330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,336 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,337 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:36,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,338 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:36,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34097321[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:36,339 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:36,339 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,341 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,341 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:36,341 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:36,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:36,341 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:36,343 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,343 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,343 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,343 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,344 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501396339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:36,345 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:36,345 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:36,345 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:36,345 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:36,345 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:36,345 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:36,347 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,347 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,347 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,348 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,349 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,356 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,356 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,356 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,356 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:36,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:36,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,356 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,357 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,357 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:36,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,357 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:36,357 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,357 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,357 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,442 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=269, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/547b096c9357440d81a558700bd62d46 2023-04-03 05:55:36,453 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/347b15ea161b456eac635380c7d2dba6 2023-04-03 05:55:36,457 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:36,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:36,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:36,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:36,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:36,462 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:36,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:36,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:36,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,463 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,463 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,464 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,464 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:36,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:36,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:36,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:36,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:36,468 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:36,468 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,468 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:36,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:36,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:36,472 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,472 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:36,472 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:36,472 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:36,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:36,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:36,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:36,474 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:36,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:36,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eaf4943[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:36,475 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:36,475 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,477 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,477 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:36,477 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:36,477 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:36,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:36,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:36,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:36,481 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:36,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:36,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:36,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:36,481 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:36,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,485 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,487 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:36,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:36,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,487 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:36,487 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:36,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,588 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:36,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:36,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:36,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:36,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:36,591 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:36,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:36,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:36,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,597 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,597 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,597 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,597 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:36,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:36,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:36,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:36,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:36,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:36,599 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,599 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:36,599 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:36,599 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:36,599 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:36,601 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:36,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:36,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6deb8e73[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:36,602 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:36,602 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,604 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:36,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:36,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:36,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,605 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:36,608 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:36,608 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:36,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:36,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:36,608 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:36,608 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:36,611 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,611 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,611 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,613 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,615 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:36,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:36,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:36,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,616 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:36,616 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:36,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501396633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501396635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501396637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,639 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501396638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,648 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:36,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501396646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,716 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:36,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:36,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:36,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:36,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:36,719 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:36,719 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,719 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:36,719 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:36,720 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,720 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,721 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,721 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,721 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,721 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:36,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:36,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:36,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:36,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:36,722 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:36,722 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,722 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:36,723 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:36,723 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:36,723 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,723 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:36,723 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:36,723 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,724 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:36,724 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:36,724 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,724 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:36,724 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:36,725 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:36,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:36,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11fb3b57[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:36,728 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:36,728 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,730 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,731 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:36,731 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:36,731 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,731 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:36,731 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:36,732 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,732 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:36,732 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,732 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:36,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:36,734 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:36,735 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:36,735 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:36,735 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:36,735 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:36,735 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:36,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,736 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,738 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,740 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,743 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,743 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,743 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,743 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,743 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,743 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,743 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:36,744 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:36,744 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,744 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,744 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,744 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:36,744 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:36,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,844 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:36,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:36,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:36,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:36,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:36,847 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:36,847 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,847 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:36,847 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:36,848 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,848 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,850 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,850 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:36,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:36,852 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:36,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:36,852 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,852 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:36,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:36,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:36,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:36,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:36,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:36,852 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,852 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:36,853 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:36,853 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,853 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:36,853 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:36,853 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,853 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:36,853 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:36,860 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,861 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,861 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,861 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:36,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e94d232[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:36,862 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:36,863 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:36,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:36,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:36,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:36,865 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,866 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:36,866 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,866 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:36,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:36,868 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:36,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:36,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:36,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:36,868 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:36,870 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/9512a77f5d884b02badc1a438fad4bc7 2023-04-03 05:55:36,870 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,870 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,870 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:36,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:36,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:36,881 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/547b096c9357440d81a558700bd62d46 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/547b096c9357440d81a558700bd62d46 2023-04-03 05:55:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,882 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:36,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,886 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/547b096c9357440d81a558700bd62d46, entries=200, sequenceid=269, filesize=32.3 K 2023-04-03 05:55:36,886 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:36,886 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:36,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:36,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:36,886 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,886 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,886 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/347b15ea161b456eac635380c7d2dba6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/347b15ea161b456eac635380c7d2dba6 2023-04-03 05:55:36,887 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:36,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:36,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:36,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:36,886 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:36,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:36,887 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:36,887 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:36,888 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:36,887 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,888 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:36,888 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:36,890 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/347b15ea161b456eac635380c7d2dba6, entries=150, sequenceid=269, filesize=11.9 K 2023-04-03 05:55:36,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/9512a77f5d884b02badc1a438fad4bc7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/9512a77f5d884b02badc1a438fad4bc7 2023-04-03 05:55:36,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/9512a77f5d884b02badc1a438fad4bc7, entries=150, sequenceid=269, filesize=11.9 K 2023-04-03 05:55:36,896 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for c534d6cb3070e98a48ea840ac1a7b2fb in 905ms, sequenceid=269, compaction requested=true 2023-04-03 05:55:36,896 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:36,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:36,896 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:36,896 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:36,896 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:36,897 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:36,897 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:36,898 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/B is initiating minor compaction (all files) 2023-04-03 05:55:36,898 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/B in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,898 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/e75065c5352d41069f228b3b8100e1c3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/64c30bf0392343f0981dfd13f74be232, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/347b15ea161b456eac635380c7d2dba6] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=35.9 K 2023-04-03 05:55:36,898 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85424 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:36,898 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/A is initiating minor compaction (all files) 2023-04-03 05:55:36,898 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/A in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,898 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/254e6015715d4b85b5da4d8ecee17bad, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/e9a282a072ce4eef92c62d77418db755, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/547b096c9357440d81a558700bd62d46] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=83.4 K 2023-04-03 05:55:36,898 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e75065c5352d41069f228b3b8100e1c3, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680501334457 2023-04-03 05:55:36,899 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 254e6015715d4b85b5da4d8ecee17bad, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680501334457 2023-04-03 05:55:36,899 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e9a282a072ce4eef92c62d77418db755, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1680501334486 2023-04-03 05:55:36,899 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 64c30bf0392343f0981dfd13f74be232, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1680501334486 2023-04-03 05:55:36,899 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 547b096c9357440d81a558700bd62d46, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680501335669 2023-04-03 05:55:36,900 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 347b15ea161b456eac635380c7d2dba6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680501335669 2023-04-03 05:55:36,911 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#B#compaction#410 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:36,936 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#A#compaction#409 average throughput is 0.70 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:36,961 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/0ec554f671334a299cf350b6b2632b75 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/0ec554f671334a299cf350b6b2632b75 2023-04-03 05:55:36,966 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/B of c534d6cb3070e98a48ea840ac1a7b2fb into 0ec554f671334a299cf350b6b2632b75(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:36,966 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:36,966 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/B, priority=13, startTime=1680501336896; duration=0sec 2023-04-03 05:55:36,966 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:36,966 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:36,967 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36804 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:36,967 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/C is initiating minor compaction (all files) 2023-04-03 05:55:36,967 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/C in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:36,967 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/f8b71582314747d5bdca77d6b6194455, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/1f8547f484104532a7db26e016876aab, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/9512a77f5d884b02badc1a438fad4bc7] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=35.9 K 2023-04-03 05:55:36,967 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting f8b71582314747d5bdca77d6b6194455, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1680501334457 2023-04-03 05:55:36,968 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1f8547f484104532a7db26e016876aab, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=249, earliestPutTs=1680501334486 2023-04-03 05:55:36,968 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 9512a77f5d884b02badc1a438fad4bc7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680501335669 2023-04-03 05:55:36,982 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#C#compaction#411 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:36,988 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:36,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:36,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:37,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:37,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:37,000 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:37,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:37,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:37,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:37,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,002 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:37,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:37,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:37,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,005 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,005 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:37,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:37,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:37,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:37,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:37,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:37,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:37,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:37,017 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/7ad454ee84c24a9092d635a601e34e90 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/7ad454ee84c24a9092d635a601e34e90 2023-04-03 05:55:37,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:37,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:37,017 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:37,017 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:37,018 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:55:37,018 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:37,018 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:37,018 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:37,018 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:37,018 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:37,020 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:37,023 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/C of c534d6cb3070e98a48ea840ac1a7b2fb into 7ad454ee84c24a9092d635a601e34e90(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:37,023 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:37,023 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/C, priority=13, startTime=1680501336896; duration=0sec 2023-04-03 05:55:37,023 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:37,034 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403834556ece10a4dfdb989ae16dc7a63ab to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403834556ece10a4dfdb989ae16dc7a63ab 2023-04-03 05:55:37,039 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=284, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/fd53e5d14ef048f48d2cc59714649a83 2023-04-03 05:55:37,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:37,143 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:37,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501397157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501397158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501397160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501397161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501397162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501397263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501397264, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501397266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501397266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501397269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,455 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/44f01adf5498413b868bd71a8c21de43 2023-04-03 05:55:37,463 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/4684bf8fbfb242e9b24f9e6a1ccd841b 2023-04-03 05:55:37,467 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/fd53e5d14ef048f48d2cc59714649a83 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/fd53e5d14ef048f48d2cc59714649a83 2023-04-03 05:55:37,470 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/fd53e5d14ef048f48d2cc59714649a83, entries=150, sequenceid=284, filesize=25.4 K 2023-04-03 05:55:37,471 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/44f01adf5498413b868bd71a8c21de43 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/44f01adf5498413b868bd71a8c21de43 2023-04-03 05:55:37,474 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/44f01adf5498413b868bd71a8c21de43, entries=150, sequenceid=284, filesize=11.9 K 2023-04-03 05:55:37,474 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/4684bf8fbfb242e9b24f9e6a1ccd841b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4684bf8fbfb242e9b24f9e6a1ccd841b 2023-04-03 05:55:37,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501397472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501397472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501397472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501397473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,477 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4684bf8fbfb242e9b24f9e6a1ccd841b, entries=150, sequenceid=284, filesize=11.9 K 2023-04-03 05:55:37,478 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for c534d6cb3070e98a48ea840ac1a7b2fb in 460ms, sequenceid=284, compaction requested=false 2023-04-03 05:55:37,478 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:37,478 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:37,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:37,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:37,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:37,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,480 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:37,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a157bf2[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:37,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:37,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:37,483 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 05:55:37,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:37,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:37,484 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:37,484 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:37,484 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:37,484 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:37,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:37,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:37,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:37,484 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,484 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:37,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:37,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:37,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:37,486 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:37,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:37,495 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:37,495 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:37,495 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:37,495 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:37,495 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:37,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:37,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:37,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:37,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:37,497 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:37,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,501 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,506 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,507 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,507 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,507 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:37,507 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403b9dda624a3484bb786859e3c6bbc0726 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b9dda624a3484bb786859e3c6bbc0726 2023-04-03 05:55:37,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:37,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:37,507 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:37,507 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:37,507 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:37,507 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:37,507 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,512 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:37,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:37,512 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,512 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:37,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:37,519 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=304, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/2d7d543743b545f6a045ccf853c0774d 2023-04-03 05:55:37,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501397542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,607 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:37,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:37,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:37,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:37,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:37,611 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:37,611 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:37,611 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:37,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:37,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,614 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:37,614 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:37,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:37,615 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,615 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:37,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:37,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:37,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:37,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:37,617 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:37,617 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:37,617 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:37,617 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:37,617 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:37,617 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:37,617 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,617 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:37,617 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:37,619 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,619 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:37,620 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:37,620 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:37,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:37,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bbce4d5[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:37,621 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:37,621 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,622 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,623 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:37,623 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:37,623 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,623 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:37,623 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:37,624 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:37,624 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:37,624 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:37,624 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:37,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:37,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:37,626 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:37,626 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:37,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:37,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:37,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:37,626 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:37,628 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,628 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:37,629 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:37,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:37,629 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,630 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,633 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,633 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,633 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:37,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:37,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:37,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:37,633 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:37,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:37,634 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:37,633 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,634 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:37,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:37,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:37,634 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,634 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,634 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,634 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501397649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,734 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:37,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:37,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:37,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:37,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:37,737 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:37,737 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:37,737 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:37,737 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:37,737 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,738 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,739 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:37,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:37,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:37,739 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,739 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,740 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:37,740 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:37,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,740 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:37,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:37,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:37,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:37,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:37,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:37,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:37,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:37,741 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:37,742 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. started... 2023-04-03 05:55:37,742 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. as already flushing 2023-04-03 05:55:37,742 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:37,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:37,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:37,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:37,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:37,743 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:37,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:37,744 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:37,744 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,744 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:37,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:37,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:37,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73555176[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:37,745 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:37,745 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,750 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,751 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:37,751 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:37,751 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:37,751 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,751 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:37,752 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,752 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:37,752 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:37,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:37,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:37,752 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:37,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:37,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:37,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:37,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:37,756 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:37,756 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:37,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:37,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:37,757 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:37,759 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,759 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:37,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,759 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:37,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:37,759 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:37,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:37,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,761 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:37,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:37,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:37,764 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:37,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:37,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:37,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,765 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:37,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:37,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:37,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:37,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:37,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:37,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:37,765 DEBUG [Thread-2670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:37,765 DEBUG [Thread-2670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:37,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:37,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501397777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501397777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501397778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501397779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,796 DEBUG [Thread-2675] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20d28b68 to 127.0.0.1:63744 2023-04-03 05:55:37,796 DEBUG [Thread-2675] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:37,797 DEBUG [Thread-2673] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x313c7ad6 to 127.0.0.1:63744 2023-04-03 05:55:37,797 DEBUG [Thread-2673] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:37,797 DEBUG [Thread-2671] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2d4bfeca to 127.0.0.1:63744 2023-04-03 05:55:37,797 DEBUG [Thread-2671] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:37,803 DEBUG [Thread-2677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x750d2a7d to 127.0.0.1:63744 2023-04-03 05:55:37,803 DEBUG [Thread-2677] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:37,804 DEBUG [Thread-2679] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27313b38 to 127.0.0.1:63744 2023-04-03 05:55:37,804 DEBUG [Thread-2679] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:37,816 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/8748469013294a0ba3fc10110c3102fc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/8748469013294a0ba3fc10110c3102fc 2023-04-03 05:55:37,819 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/A of c534d6cb3070e98a48ea840ac1a7b2fb into 8748469013294a0ba3fc10110c3102fc(size=25.5 K), total size for store is 51.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:37,819 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:37,819 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/A, priority=13, startTime=1680501336896; duration=0sec 2023-04-03 05:55:37,819 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:37,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:37,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501397853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:37,865 DEBUG [Thread-2670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:37,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:37,928 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/8f95e331bcac401cbb1e0851a9d98d0a 2023-04-03 05:55:38,157 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:38,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36600 deadline: 1680501398157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:38,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:38,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36636 deadline: 1680501398285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:38,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:38,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36616 deadline: 1680501398286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:38,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:38,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36654 deadline: 1680501398287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:38,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:38,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36580 deadline: 1680501398287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:38,337 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=304 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/e852f70dfb274602957cac76734257ce 2023-04-03 05:55:38,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/2d7d543743b545f6a045ccf853c0774d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/2d7d543743b545f6a045ccf853c0774d 2023-04-03 05:55:38,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/2d7d543743b545f6a045ccf853c0774d, entries=200, sequenceid=304, filesize=32.3 K 2023-04-03 05:55:38,343 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/8f95e331bcac401cbb1e0851a9d98d0a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/8f95e331bcac401cbb1e0851a9d98d0a 2023-04-03 05:55:38,346 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/8f95e331bcac401cbb1e0851a9d98d0a, entries=150, sequenceid=304, filesize=11.9 K 2023-04-03 05:55:38,346 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/e852f70dfb274602957cac76734257ce as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/e852f70dfb274602957cac76734257ce 2023-04-03 05:55:38,349 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/e852f70dfb274602957cac76734257ce, entries=150, sequenceid=304, filesize=11.9 K 2023-04-03 05:55:38,349 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for c534d6cb3070e98a48ea840ac1a7b2fb in 866ms, sequenceid=304, compaction requested=true 2023-04-03 05:55:38,349 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:38,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:38,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:38,350 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:38,350 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:38,350 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:38,350 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85281 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:38,350 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36661 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:38,350 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/A is initiating minor compaction (all files) 2023-04-03 05:55:38,350 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/B is initiating minor compaction (all files) 2023-04-03 05:55:38,351 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/A in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:38,351 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/B in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:38,351 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/8748469013294a0ba3fc10110c3102fc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/fd53e5d14ef048f48d2cc59714649a83, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/2d7d543743b545f6a045ccf853c0774d] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=83.3 K 2023-04-03 05:55:38,351 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/0ec554f671334a299cf350b6b2632b75, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/44f01adf5498413b868bd71a8c21de43, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/8f95e331bcac401cbb1e0851a9d98d0a] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=35.8 K 2023-04-03 05:55:38,351 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8748469013294a0ba3fc10110c3102fc, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680501335669 2023-04-03 05:55:38,351 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0ec554f671334a299cf350b6b2632b75, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680501335669 2023-04-03 05:55:38,351 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting fd53e5d14ef048f48d2cc59714649a83, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680501336013 2023-04-03 05:55:38,351 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 44f01adf5498413b868bd71a8c21de43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680501336013 2023-04-03 05:55:38,351 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 2d7d543743b545f6a045ccf853c0774d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680501337159 2023-04-03 05:55:38,351 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 8f95e331bcac401cbb1e0851a9d98d0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680501337159 2023-04-03 05:55:38,356 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#B#compaction#418 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:38,359 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#A#compaction#419 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:38,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:38,662 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-03 05:55:38,662 DEBUG [Thread-2664] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x14e151fb to 127.0.0.1:63744 2023-04-03 05:55:38,662 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:38,662 DEBUG [Thread-2664] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:38,663 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:38,663 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:38,663 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:38,663 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:38,663 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:38,764 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/0ccd49ff85a14ce7ac26ec7fdb1bbdfd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/0ccd49ff85a14ce7ac26ec7fdb1bbdfd 2023-04-03 05:55:38,769 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/B of c534d6cb3070e98a48ea840ac1a7b2fb into 0ccd49ff85a14ce7ac26ec7fdb1bbdfd(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:38,769 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:38,769 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/B, priority=13, startTime=1680501338350; duration=0sec 2023-04-03 05:55:38,770 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:38,770 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:38,770 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36661 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:38,770 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): c534d6cb3070e98a48ea840ac1a7b2fb/C is initiating minor compaction (all files) 2023-04-03 05:55:38,770 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of c534d6cb3070e98a48ea840ac1a7b2fb/C in TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:38,771 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/7ad454ee84c24a9092d635a601e34e90, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4684bf8fbfb242e9b24f9e6a1ccd841b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/e852f70dfb274602957cac76734257ce] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp, totalSize=35.8 K 2023-04-03 05:55:38,771 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 7ad454ee84c24a9092d635a601e34e90, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1680501335669 2023-04-03 05:55:38,771 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 4684bf8fbfb242e9b24f9e6a1ccd841b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680501336013 2023-04-03 05:55:38,771 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e852f70dfb274602957cac76734257ce, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=304, earliestPutTs=1680501337159 2023-04-03 05:55:38,776 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): c534d6cb3070e98a48ea840ac1a7b2fb#C#compaction#421 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:39,072 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403210078147d6b4214a53f70414fd04ce1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403210078147d6b4214a53f70414fd04ce1 2023-04-03 05:55:39,182 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/db609388ba154c878ffb2b3a9ed647a0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/db609388ba154c878ffb2b3a9ed647a0 2023-04-03 05:55:39,185 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/C of c534d6cb3070e98a48ea840ac1a7b2fb into db609388ba154c878ffb2b3a9ed647a0(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:39,185 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:39,185 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/C, priority=13, startTime=1680501338350; duration=0sec 2023-04-03 05:55:39,185 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:39,291 DEBUG [Thread-2668] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x722b2407 to 127.0.0.1:63744 2023-04-03 05:55:39,292 DEBUG [Thread-2668] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:39,292 DEBUG [Thread-2660] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1cee66b5 to 127.0.0.1:63744 2023-04-03 05:55:39,293 DEBUG [Thread-2660] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:39,295 DEBUG [Thread-2662] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4c1d19b3 to 127.0.0.1:63744 2023-04-03 05:55:39,295 DEBUG [Thread-2662] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:39,297 DEBUG [Thread-2666] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x455a88b1 to 127.0.0.1:63744 2023-04-03 05:55:39,297 DEBUG [Thread-2666] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 45 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 61 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 47 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 45 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 1800 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 5400 rows 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 1809 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 5426 rows 2023-04-03 05:55:39,297 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 1811 2023-04-03 05:55:39,298 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 5433 rows 2023-04-03 05:55:39,298 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 1800 2023-04-03 05:55:39,298 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 5400 rows 2023-04-03 05:55:39,298 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 1803 2023-04-03 05:55:39,298 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 5409 rows 2023-04-03 05:55:39,298 INFO [Listener at localhost/37275] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 05:55:39,298 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67b6d133 to 127.0.0.1:63744 2023-04-03 05:55:39,298 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:55:39,301 INFO [Listener at localhost/37275] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 05:55:39,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-03 05:55:39,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:39,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 05:55:39,306 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501339306"}]},"ts":"1680501339306"} 2023-04-03 05:55:39,306 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 05:55:39,308 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 05:55:39,308 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c534d6cb3070e98a48ea840ac1a7b2fb, UNASSIGN}] 2023-04-03 05:55:39,309 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=c534d6cb3070e98a48ea840ac1a7b2fb, UNASSIGN 2023-04-03 05:55:39,310 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=c534d6cb3070e98a48ea840ac1a7b2fb, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:39,310 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501339310"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501339310"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501339310"}]},"ts":"1680501339310"} 2023-04-03 05:55:39,311 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:55:39,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 05:55:39,462 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:39,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing c534d6cb3070e98a48ea840ac1a7b2fb, disabling compactions & flushes 2023-04-03 05:55:39,462 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 1 compactions & cache flush to complete for region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:39,477 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=319, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/cf21064003c24bfc8d5f58232a13009d 2023-04-03 05:55:39,576 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/53678e4bb3d648ef95390a4176c32bfa as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/53678e4bb3d648ef95390a4176c32bfa 2023-04-03 05:55:39,579 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in c534d6cb3070e98a48ea840ac1a7b2fb/A of c534d6cb3070e98a48ea840ac1a7b2fb into 53678e4bb3d648ef95390a4176c32bfa(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-03 05:55:39,579 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:39,579 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:39,580 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb., storeName=c534d6cb3070e98a48ea840ac1a7b2fb/A, priority=13, startTime=1680501338349; duration=1sec 2023-04-03 05:55:39,580 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:39,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 05:55:39,885 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/75acb153d6114b5590b48562d94307dd 2023-04-03 05:55:39,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 05:55:40,293 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=319 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/4681132da7134658a0f0ab7218c11fcc 2023-04-03 05:55:40,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/cf21064003c24bfc8d5f58232a13009d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/cf21064003c24bfc8d5f58232a13009d 2023-04-03 05:55:40,299 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/cf21064003c24bfc8d5f58232a13009d, entries=150, sequenceid=319, filesize=25.4 K 2023-04-03 05:55:40,299 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/75acb153d6114b5590b48562d94307dd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/75acb153d6114b5590b48562d94307dd 2023-04-03 05:55:40,302 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/75acb153d6114b5590b48562d94307dd, entries=150, sequenceid=319, filesize=11.9 K 2023-04-03 05:55:40,303 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/4681132da7134658a0f0ab7218c11fcc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4681132da7134658a0f0ab7218c11fcc 2023-04-03 05:55:40,305 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4681132da7134658a0f0ab7218c11fcc, entries=150, sequenceid=319, filesize=11.9 K 2023-04-03 05:55:40,306 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=26.84 KB/27480 for c534d6cb3070e98a48ea840ac1a7b2fb in 1644ms, sequenceid=319, compaction requested=false 2023-04-03 05:55:40,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:40,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:40,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:40,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. after waiting 0 ms 2023-04-03 05:55:40,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:40,306 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing c534d6cb3070e98a48ea840ac1a7b2fb 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-03 05:55:40,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=A 2023-04-03 05:55:40,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:40,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=B 2023-04-03 05:55:40,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:40,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK c534d6cb3070e98a48ea840ac1a7b2fb, store=C 2023-04-03 05:55:40,306 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:40,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 05:55:40,715 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403af24240e4b1a4a87b78101fb84a2d32f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403af24240e4b1a4a87b78101fb84a2d32f 2023-04-03 05:55:41,120 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=329, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/15268d84b84c41d780fd7ab62e52828a 2023-04-03 05:55:41,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 05:55:41,528 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/e4193d9297fb47798694be3a251bc2a2 2023-04-03 05:55:41,935 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=329 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/4e6ec297c816446a888b1ae01cb75d94 2023-04-03 05:55:41,939 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/A/15268d84b84c41d780fd7ab62e52828a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/15268d84b84c41d780fd7ab62e52828a 2023-04-03 05:55:41,941 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/15268d84b84c41d780fd7ab62e52828a, entries=150, sequenceid=329, filesize=25.4 K 2023-04-03 05:55:41,942 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/B/e4193d9297fb47798694be3a251bc2a2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/e4193d9297fb47798694be3a251bc2a2 2023-04-03 05:55:41,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/e4193d9297fb47798694be3a251bc2a2, entries=150, sequenceid=329, filesize=11.9 K 2023-04-03 05:55:41,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/.tmp/C/4e6ec297c816446a888b1ae01cb75d94 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4e6ec297c816446a888b1ae01cb75d94 2023-04-03 05:55:41,948 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4e6ec297c816446a888b1ae01cb75d94, entries=150, sequenceid=329, filesize=11.9 K 2023-04-03 05:55:41,949 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for c534d6cb3070e98a48ea840ac1a7b2fb in 1643ms, sequenceid=329, compaction requested=true 2023-04-03 05:55:41,949 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/254e6015715d4b85b5da4d8ecee17bad, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/e9a282a072ce4eef92c62d77418db755, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/547b096c9357440d81a558700bd62d46, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/8748469013294a0ba3fc10110c3102fc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/fd53e5d14ef048f48d2cc59714649a83, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/2d7d543743b545f6a045ccf853c0774d] to archive 2023-04-03 05:55:41,950 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:55:41,951 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/254e6015715d4b85b5da4d8ecee17bad to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/254e6015715d4b85b5da4d8ecee17bad 2023-04-03 05:55:41,952 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/e9a282a072ce4eef92c62d77418db755 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/e9a282a072ce4eef92c62d77418db755 2023-04-03 05:55:41,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/547b096c9357440d81a558700bd62d46 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/547b096c9357440d81a558700bd62d46 2023-04-03 05:55:41,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/8748469013294a0ba3fc10110c3102fc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/8748469013294a0ba3fc10110c3102fc 2023-04-03 05:55:41,954 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/fd53e5d14ef048f48d2cc59714649a83 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/fd53e5d14ef048f48d2cc59714649a83 2023-04-03 05:55:41,955 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/2d7d543743b545f6a045ccf853c0774d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/2d7d543743b545f6a045ccf853c0774d 2023-04-03 05:55:41,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/e75065c5352d41069f228b3b8100e1c3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/64c30bf0392343f0981dfd13f74be232, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/0ec554f671334a299cf350b6b2632b75, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/347b15ea161b456eac635380c7d2dba6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/44f01adf5498413b868bd71a8c21de43, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/8f95e331bcac401cbb1e0851a9d98d0a] to archive 2023-04-03 05:55:41,956 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:55:41,957 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/e75065c5352d41069f228b3b8100e1c3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/e75065c5352d41069f228b3b8100e1c3 2023-04-03 05:55:41,958 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/64c30bf0392343f0981dfd13f74be232 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/64c30bf0392343f0981dfd13f74be232 2023-04-03 05:55:41,958 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/0ec554f671334a299cf350b6b2632b75 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/0ec554f671334a299cf350b6b2632b75 2023-04-03 05:55:41,959 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/347b15ea161b456eac635380c7d2dba6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/347b15ea161b456eac635380c7d2dba6 2023-04-03 05:55:41,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/44f01adf5498413b868bd71a8c21de43 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/44f01adf5498413b868bd71a8c21de43 2023-04-03 05:55:41,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/8f95e331bcac401cbb1e0851a9d98d0a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/8f95e331bcac401cbb1e0851a9d98d0a 2023-04-03 05:55:41,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/f8b71582314747d5bdca77d6b6194455, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/1f8547f484104532a7db26e016876aab, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/7ad454ee84c24a9092d635a601e34e90, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/9512a77f5d884b02badc1a438fad4bc7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4684bf8fbfb242e9b24f9e6a1ccd841b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/e852f70dfb274602957cac76734257ce] to archive 2023-04-03 05:55:41,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:55:41,963 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/f8b71582314747d5bdca77d6b6194455 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/f8b71582314747d5bdca77d6b6194455 2023-04-03 05:55:41,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/1f8547f484104532a7db26e016876aab to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/1f8547f484104532a7db26e016876aab 2023-04-03 05:55:41,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/7ad454ee84c24a9092d635a601e34e90 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/7ad454ee84c24a9092d635a601e34e90 2023-04-03 05:55:41,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/9512a77f5d884b02badc1a438fad4bc7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/9512a77f5d884b02badc1a438fad4bc7 2023-04-03 05:55:41,966 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4684bf8fbfb242e9b24f9e6a1ccd841b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4684bf8fbfb242e9b24f9e6a1ccd841b 2023-04-03 05:55:41,966 DEBUG [StoreCloser-TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/e852f70dfb274602957cac76734257ce to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/e852f70dfb274602957cac76734257ce 2023-04-03 05:55:41,969 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/recovered.edits/332.seqid, newMaxSeqId=332, maxSeqId=4 2023-04-03 05:55:41,970 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb. 2023-04-03 05:55:41,970 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for c534d6cb3070e98a48ea840ac1a7b2fb: 2023-04-03 05:55:41,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:41,972 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=c534d6cb3070e98a48ea840ac1a7b2fb, regionState=CLOSED 2023-04-03 05:55:41,972 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501341972"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501341972"}]},"ts":"1680501341972"} 2023-04-03 05:55:41,974 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-03 05:55:41,974 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure c534d6cb3070e98a48ea840ac1a7b2fb, server=jenkins-hbase4.apache.org,37397,1680501214665 in 2.6620 sec 2023-04-03 05:55:41,975 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-03 05:55:41,975 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=c534d6cb3070e98a48ea840ac1a7b2fb, UNASSIGN in 2.6660 sec 2023-04-03 05:55:41,976 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501341975"}]},"ts":"1680501341975"} 2023-04-03 05:55:41,976 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 05:55:41,977 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 05:55:41,979 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.6760 sec 2023-04-03 05:55:43,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-03 05:55:43,411 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-03 05:55:43,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-03 05:55:43,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:43,413 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:43,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-03 05:55:43,414 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:43,418 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:43,420 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/recovered.edits] 2023-04-03 05:55:43,422 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/15268d84b84c41d780fd7ab62e52828a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/15268d84b84c41d780fd7ab62e52828a 2023-04-03 05:55:43,423 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/53678e4bb3d648ef95390a4176c32bfa to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/53678e4bb3d648ef95390a4176c32bfa 2023-04-03 05:55:43,423 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/cf21064003c24bfc8d5f58232a13009d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/A/cf21064003c24bfc8d5f58232a13009d 2023-04-03 05:55:43,425 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/0ccd49ff85a14ce7ac26ec7fdb1bbdfd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/0ccd49ff85a14ce7ac26ec7fdb1bbdfd 2023-04-03 05:55:43,426 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/75acb153d6114b5590b48562d94307dd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/75acb153d6114b5590b48562d94307dd 2023-04-03 05:55:43,427 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/e4193d9297fb47798694be3a251bc2a2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/B/e4193d9297fb47798694be3a251bc2a2 2023-04-03 05:55:43,429 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4681132da7134658a0f0ab7218c11fcc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4681132da7134658a0f0ab7218c11fcc 2023-04-03 05:55:43,430 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4e6ec297c816446a888b1ae01cb75d94 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/4e6ec297c816446a888b1ae01cb75d94 2023-04-03 05:55:43,431 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/db609388ba154c878ffb2b3a9ed647a0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/C/db609388ba154c878ffb2b3a9ed647a0 2023-04-03 05:55:43,432 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/recovered.edits/332.seqid to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb/recovered.edits/332.seqid 2023-04-03 05:55:43,433 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/c534d6cb3070e98a48ea840ac1a7b2fb 2023-04-03 05:55:43,433 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 05:55:43,433 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 05:55:43,434 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-03 05:55:43,436 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040302df828bfb56403ea0b9811252f48c57 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040302df828bfb56403ea0b9811252f48c57 2023-04-03 05:55:43,436 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031a5fd1e411b44a9495d9969b0da89332 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031a5fd1e411b44a9495d9969b0da89332 2023-04-03 05:55:43,437 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403210078147d6b4214a53f70414fd04ce1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403210078147d6b4214a53f70414fd04ce1 2023-04-03 05:55:43,438 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032498124a1b444e4894a755f92f0b1aa5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032498124a1b444e4894a755f92f0b1aa5 2023-04-03 05:55:43,439 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032ea6ac149c0b4f61857ca87a87d42a38 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032ea6ac149c0b4f61857ca87a87d42a38 2023-04-03 05:55:43,440 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040354a6a545b565468693eb6f4c003816a5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040354a6a545b565468693eb6f4c003816a5 2023-04-03 05:55:43,441 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040364ead20bbc204c7f827b997b32b0b60e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040364ead20bbc204c7f827b997b32b0b60e 2023-04-03 05:55:43,441 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037c201ae8e34f46c5983eadaa1e548440 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037c201ae8e34f46c5983eadaa1e548440 2023-04-03 05:55:43,442 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403834556ece10a4dfdb989ae16dc7a63ab to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403834556ece10a4dfdb989ae16dc7a63ab 2023-04-03 05:55:43,443 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038450ef5478874bd5bfd841c4d85b549c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304038450ef5478874bd5bfd841c4d85b549c 2023-04-03 05:55:43,444 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a768688a637043f895963b326e84115e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403a768688a637043f895963b326e84115e 2023-04-03 05:55:43,445 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403af24240e4b1a4a87b78101fb84a2d32f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403af24240e4b1a4a87b78101fb84a2d32f 2023-04-03 05:55:43,446 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b128963db0d749748c83d9814d30ee38 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b128963db0d749748c83d9814d30ee38 2023-04-03 05:55:43,447 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b9dda624a3484bb786859e3c6bbc0726 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b9dda624a3484bb786859e3c6bbc0726 2023-04-03 05:55:43,448 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c5376c82c6674267b13abb533d2e2270 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c5376c82c6674267b13abb533d2e2270 2023-04-03 05:55:43,449 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d955865e1ec34a8c9b0af855658a66e8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d955865e1ec34a8c9b0af855658a66e8 2023-04-03 05:55:43,450 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e1d497cf0cfb4af18898763ab8ba4a96 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e1d497cf0cfb4af18898763ab8ba4a96 2023-04-03 05:55:43,451 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e3c4345e783e428ead4b0c5e714128ec to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e3c4345e783e428ead4b0c5e714128ec 2023-04-03 05:55:43,452 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f4795502cbe746dca64035f18ff7a396 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f4795502cbe746dca64035f18ff7a396 2023-04-03 05:55:43,453 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f578b1143b48475893ba5440c1382b9d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403f578b1143b48475893ba5440c1382b9d 2023-04-03 05:55:43,453 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 05:55:43,455 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:43,456 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 05:55:43,458 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 05:55:43,459 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:43,459 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 05:55:43,459 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680501343459"}]},"ts":"9223372036854775807"} 2023-04-03 05:55:43,460 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 05:55:43,460 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => c534d6cb3070e98a48ea840ac1a7b2fb, NAME => 'TestAcidGuarantees,,1680501315725.c534d6cb3070e98a48ea840ac1a7b2fb.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 05:55:43,460 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 05:55:43,460 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680501343460"}]},"ts":"9223372036854775807"} 2023-04-03 05:55:43,462 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 05:55:43,466 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:43,467 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 55 msec 2023-04-03 05:55:43,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-03 05:55:43,515 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-03 05:55:43,526 INFO [Listener at localhost/37275] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=230 (was 224) - Thread LEAK? -, OpenFileDescriptor=541 (was 525) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=438 (was 318) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=3353 (was 3387) 2023-04-03 05:55:43,537 INFO [Listener at localhost/37275] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=230, OpenFileDescriptor=541, MaxFileDescriptor=60000, SystemLoadAverage=438, ProcessCount=169, AvailableMemoryMB=3353 2023-04-03 05:55:43,539 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 05:55:43,539 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 05:55:43,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 05:55:43,541 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 05:55:43,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-03 05:55:43,542 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 05:55:43,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 05:55:43,543 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:43,543 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a empty. 2023-04-03 05:55:43,543 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:43,543 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 05:55:43,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 05:55:43,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 05:55:43,950 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 05:55:43,951 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5d777b61781e184e82ec86ca4072f25a, NAME => 'TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp 2023-04-03 05:55:43,956 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:55:43,956 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5d777b61781e184e82ec86ca4072f25a, disabling compactions & flushes 2023-04-03 05:55:43,956 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:43,956 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:43,956 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. after waiting 0 ms 2023-04-03 05:55:43,956 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:43,956 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:43,956 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:43,958 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 05:55:43,959 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501343959"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501343959"}]},"ts":"1680501343959"} 2023-04-03 05:55:43,960 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 05:55:43,960 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 05:55:43,961 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501343961"}]},"ts":"1680501343961"} 2023-04-03 05:55:43,961 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 05:55:43,964 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5d777b61781e184e82ec86ca4072f25a, ASSIGN}] 2023-04-03 05:55:43,965 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5d777b61781e184e82ec86ca4072f25a, ASSIGN 2023-04-03 05:55:43,966 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5d777b61781e184e82ec86ca4072f25a, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37397,1680501214665; forceNewPlan=false, retain=false 2023-04-03 05:55:44,117 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=5d777b61781e184e82ec86ca4072f25a, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,117 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501344117"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501344117"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501344117"}]},"ts":"1680501344117"} 2023-04-03 05:55:44,118 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:55:44,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 05:55:44,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:44,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5d777b61781e184e82ec86ca4072f25a, NAME => 'TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.', STARTKEY => '', ENDKEY => ''} 2023-04-03 05:55:44,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:44,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:55:44,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:44,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:44,274 INFO [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:44,275 DEBUG [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A 2023-04-03 05:55:44,275 DEBUG [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A 2023-04-03 05:55:44,275 INFO [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:55:44,275 INFO [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5d777b61781e184e82ec86ca4072f25a columnFamilyName A 2023-04-03 05:55:44,276 INFO [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] regionserver.HStore(310): Store=5d777b61781e184e82ec86ca4072f25a/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:55:44,276 INFO [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:44,277 DEBUG [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B 2023-04-03 05:55:44,277 DEBUG [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B 2023-04-03 05:55:44,277 INFO [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:55:44,277 INFO [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5d777b61781e184e82ec86ca4072f25a columnFamilyName B 2023-04-03 05:55:44,278 INFO [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] regionserver.HStore(310): Store=5d777b61781e184e82ec86ca4072f25a/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:55:44,278 INFO [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:44,278 DEBUG [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C 2023-04-03 05:55:44,279 DEBUG [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C 2023-04-03 05:55:44,279 INFO [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:55:44,279 INFO [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5d777b61781e184e82ec86ca4072f25a columnFamilyName C 2023-04-03 05:55:44,279 INFO [StoreOpener-5d777b61781e184e82ec86ca4072f25a-1] regionserver.HStore(310): Store=5d777b61781e184e82ec86ca4072f25a/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:55:44,279 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:44,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:44,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:44,281 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 05:55:44,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:44,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 05:55:44,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5d777b61781e184e82ec86ca4072f25a; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62658400, jitterRate=-0.06631708145141602}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 05:55:44,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:44,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., pid=49, masterSystemTime=1680501344269 2023-04-03 05:55:44,285 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:44,286 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:44,286 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=5d777b61781e184e82ec86ca4072f25a, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,286 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501344286"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680501344286"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680501344286"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680501344286"}]},"ts":"1680501344286"} 2023-04-03 05:55:44,288 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-03 05:55:44,288 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 in 169 msec 2023-04-03 05:55:44,289 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-03 05:55:44,289 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5d777b61781e184e82ec86ca4072f25a, ASSIGN in 324 msec 2023-04-03 05:55:44,289 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 05:55:44,289 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501344289"}]},"ts":"1680501344289"} 2023-04-03 05:55:44,290 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 05:55:44,292 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 05:55:44,292 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 753 msec 2023-04-03 05:55:44,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-03 05:55:44,645 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-03 05:55:44,646 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x66369677 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:44,651 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1df28b68, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:44,652 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:44,654 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34570, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:44,655 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 05:55:44,657 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:55824, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 05:55:44,658 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x6e0b6187 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:44,662 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c5cd71f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:44,662 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x2ab56e54 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:44,666 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1f0735dc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:44,667 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x15840296 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:44,673 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14c209bc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:44,673 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x3816bdf5 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:44,677 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@db388d3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:44,677 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x50150525 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:44,680 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5de1e3c7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:44,681 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x7a4e3b2e to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:44,684 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28e3107b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:44,684 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x1f65627a to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:44,691 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@14f9adf3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:44,692 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x15f74c78 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:44,695 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21359a0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:44,695 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x1401cac1 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:44,699 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3bbac0b5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:44,700 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x1f6e865f to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:55:44,703 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a300b1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:55:44,706 DEBUG [hconnection-0x2149da82-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:44,708 DEBUG [hconnection-0xc0cafc5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:44,710 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34586, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:44,710 DEBUG [hconnection-0x6c56a5fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:44,711 DEBUG [hconnection-0x178113fe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:44,711 DEBUG [hconnection-0x8684a7c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:44,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:44,712 DEBUG [hconnection-0x3d5da49e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:44,713 DEBUG [hconnection-0x4acea22f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:44,713 DEBUG [hconnection-0x46305e9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:44,713 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34598, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:44,714 DEBUG [hconnection-0x7a231dea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:44,716 DEBUG [hconnection-0xa5a7a8f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:55:44,720 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34610, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:44,720 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34608, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:44,721 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34638, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:44,720 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34600, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:44,721 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34622, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:44,721 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34614, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:44,721 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34662, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:44,721 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:34648, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:55:44,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:44,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:44,724 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:44,724 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:44,724 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:44,725 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:44,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,727 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:44,730 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:44,730 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:44,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:44,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:44,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:44,732 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:55:44,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:44,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:44,733 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:44,734 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:44,734 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:44,734 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:44,734 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:44,734 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:44,734 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:44,734 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,734 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:44,741 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:44,741 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:44,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:44,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:44,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:44,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:44,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:44,743 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:44,743 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:44,743 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:44,743 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:44,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:44,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:44,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:44,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:44,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:44,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501404742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:44,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501404742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,744 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:44,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501404742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680501344737, max timestamp=1680501344742Num uniques 859; 2023-04-03 05:55:44,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:55:44,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680501344737, max timestamp=1680501344742Num uniques 859; 2023-04-03 05:55:44,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:55:44,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680501344737, max timestamp=1680501344742Num uniques 859; 2023-04-03 05:55:44,745 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:55:44,746 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,746 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:44,746 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,747 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:44,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:44,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:44,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:44,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b18186c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:44,749 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:44,749 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,750 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:44,750 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:44,750 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,750 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:44,751 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/9cea537836f14288a89cbd2207ee44d0 2023-04-03 05:55:44,754 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:44,754 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:44,754 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:44,754 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:44,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:44,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:44,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:44,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:44,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:44,762 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:44,762 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:44,762 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:44,762 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:44,762 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:44,762 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:44,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:44,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:44,766 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:44,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,767 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,768 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:44,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:44,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:44,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:44,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:44,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,770 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:44,770 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:44,770 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:44,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:44,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:44,770 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:44,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:44,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:44,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:44,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:44,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:44,777 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/a2fb9ad8a04449f5a43f02f3bdee0d19 2023-04-03 05:55:44,782 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:44,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501404777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:44,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501404778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:44,849 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501404845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,854 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:44,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501404848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:44,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501404852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,870 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:44,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:44,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:44,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:44,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:44,876 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:44,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:44,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:44,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:44,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,877 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:44,878 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,878 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:44,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:44,878 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,878 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:44,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:44,878 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:44,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:44,879 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:44,879 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:44,879 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:44,879 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:44,879 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:44,880 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:44,880 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:44,880 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:44,880 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:44,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:44,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,884 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:44,885 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:44,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501404883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:44,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:44,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501404884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:44,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:44,887 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1af33e16[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:44,887 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:44,887 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,888 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,888 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:44,888 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:44,888 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,888 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:44,888 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:44,889 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,889 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:44,889 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:44,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:44,889 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:44,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:44,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:44,891 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:44,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:44,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:44,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:44,892 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:44,893 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,893 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:44,893 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,893 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:44,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:44,893 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:44,894 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:44,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:44,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:44,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:44,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:44,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:44,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:44,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:44,897 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:44,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:44,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:44,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:44,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:44,898 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:44,898 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:44,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:44,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:44,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:44,998 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:44,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:45,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:45,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:45,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:45,001 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:45,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:45,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:45,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:45,002 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,002 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,003 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:45,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:45,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:45,008 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:45,008 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:45,008 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:45,008 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:45,008 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:45,012 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:45,012 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:45,012 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:45,012 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:45,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,013 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:45,013 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:45,014 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,014 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:45,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:45,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4635ebd1[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:45,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:45,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,017 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:45,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,018 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:45,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:45,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:45,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:45,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:45,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:45,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:45,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:45,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:45,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:45,023 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:45,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:45,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:45,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:45,023 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:45,024 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,024 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:45,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,025 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:45,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:45,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:45,025 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,026 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,028 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:45,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:45,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:45,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:45,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:45,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:45,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:45,029 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:45,029 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:45,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:45,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:45,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501405050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501405055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501405059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,094 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501405087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501405087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,129 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:45,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:45,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:45,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:45,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:45,133 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:45,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:45,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:45,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:45,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,134 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:45,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:45,134 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,135 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,135 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:45,135 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,136 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:45,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:45,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:45,136 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:45,137 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:45,137 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:45,137 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:45,137 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:45,139 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,139 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:45,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:45,140 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:45,140 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:45,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:45,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f32dc80[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:45,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:45,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,142 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:45,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:45,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:45,144 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:45,144 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:45,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:45,147 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:45,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:45,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:45,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:45,147 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:45,148 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,148 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:45,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:45,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,148 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:45,149 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,150 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:45,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,153 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:45,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:45,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:45,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:45,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:45,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:45,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,154 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:45,154 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:45,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:45,154 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:45,222 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=12 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/aa7f32cf79de4496af21c86790e76bc5 2023-04-03 05:55:45,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/9cea537836f14288a89cbd2207ee44d0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9cea537836f14288a89cbd2207ee44d0 2023-04-03 05:55:45,230 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9cea537836f14288a89cbd2207ee44d0, entries=150, sequenceid=12, filesize=11.6 K 2023-04-03 05:55:45,230 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/a2fb9ad8a04449f5a43f02f3bdee0d19 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/a2fb9ad8a04449f5a43f02f3bdee0d19 2023-04-03 05:55:45,234 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/a2fb9ad8a04449f5a43f02f3bdee0d19, entries=150, sequenceid=12, filesize=11.6 K 2023-04-03 05:55:45,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/aa7f32cf79de4496af21c86790e76bc5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/aa7f32cf79de4496af21c86790e76bc5 2023-04-03 05:55:45,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/aa7f32cf79de4496af21c86790e76bc5, entries=150, sequenceid=12, filesize=11.6 K 2023-04-03 05:55:45,238 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 5d777b61781e184e82ec86ca4072f25a in 506ms, sequenceid=12, compaction requested=false 2023-04-03 05:55:45,238 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:45,254 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:45,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:45,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:45,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:45,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:45,257 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:45,257 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:45,257 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:45,257 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:45,258 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,258 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,259 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:45,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:45,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:45,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,259 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:45,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:45,260 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:45,260 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:45,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:45,260 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:45,261 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:45,261 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-03 05:55:45,261 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:45,261 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:45,261 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:45,261 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:45,261 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:45,261 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:45,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:45,359 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:45,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501405373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501405374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501405377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501405397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501405397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501405483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501405484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501405484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,671 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/3394f69cdd6946c38e4b2998daf786d8 2023-04-03 05:55:45,683 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/b365c783d3fe4702b5254b51b6bbf9ee 2023-04-03 05:55:45,694 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/2e65cae376d84ffcb2ab8f8e471b1ed8 2023-04-03 05:55:45,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501405691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/3394f69cdd6946c38e4b2998daf786d8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/3394f69cdd6946c38e4b2998daf786d8 2023-04-03 05:55:45,697 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501405692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,698 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501405692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,700 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/3394f69cdd6946c38e4b2998daf786d8, entries=150, sequenceid=46, filesize=11.6 K 2023-04-03 05:55:45,701 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/b365c783d3fe4702b5254b51b6bbf9ee as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b365c783d3fe4702b5254b51b6bbf9ee 2023-04-03 05:55:45,704 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b365c783d3fe4702b5254b51b6bbf9ee, entries=150, sequenceid=46, filesize=11.6 K 2023-04-03 05:55:45,705 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/2e65cae376d84ffcb2ab8f8e471b1ed8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/2e65cae376d84ffcb2ab8f8e471b1ed8 2023-04-03 05:55:45,708 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/2e65cae376d84ffcb2ab8f8e471b1ed8, entries=150, sequenceid=46, filesize=11.6 K 2023-04-03 05:55:45,709 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 5d777b61781e184e82ec86ca4072f25a in 448ms, sequenceid=46, compaction requested=false 2023-04-03 05:55:45,709 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:45,709 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:45,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:45,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:45,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:45,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:45,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:45,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64126c72[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:45,713 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:45,713 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,714 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,714 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:45,714 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:45,714 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:45,714 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,714 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:45,723 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,723 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:45,723 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:45,724 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:45,726 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:45,726 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:45,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:45,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:45,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:45,726 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:45,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:45,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,727 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:45,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:45,728 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:45,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,729 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:45,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:45,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,731 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,731 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,731 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:45,731 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:45,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:45,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:45,731 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:45,731 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:45,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:45,731 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,731 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,731 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:45,731 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,732 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:45,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:45,732 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:45,833 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:45,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:45,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:45,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:45,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:45,838 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:45,838 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:45,838 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:45,838 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:45,838 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,839 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,840 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:45,840 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,840 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,840 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:45,841 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:45,841 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:45,841 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:45,841 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:45,841 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:45,841 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:45,841 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:45,842 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:45,842 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:45,842 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:45,842 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:45,842 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 05:55:45,842 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:45,842 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:45,842 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:45,842 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:45,842 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:45,842 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:45,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:45,904 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:45,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:45,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501405963, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:45,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501345912, max timestamp=1680501345968Num uniques 859; 2023-04-03 05:55:45,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:55:45,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501345912, max timestamp=1680501345968Num uniques 859; 2023-04-03 05:55:45,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:55:45,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501345912, max timestamp=1680501345968Num uniques 859; 2023-04-03 05:55:45,970 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:55:46,050 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501406028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,062 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501406050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501406050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501406050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501406071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,158 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501406151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501406163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501406164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,171 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501406164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,266 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/730488298f7c4a1ab323a7f83ce2bd01 2023-04-03 05:55:46,276 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/b32e89ffe9d64c5191ad9bf896098363 2023-04-03 05:55:46,281 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501406278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,287 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/312d7ba753034b968e50d8da1e67d9ff 2023-04-03 05:55:46,290 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/730488298f7c4a1ab323a7f83ce2bd01 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/730488298f7c4a1ab323a7f83ce2bd01 2023-04-03 05:55:46,293 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/730488298f7c4a1ab323a7f83ce2bd01, entries=150, sequenceid=57, filesize=11.6 K 2023-04-03 05:55:46,294 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/b32e89ffe9d64c5191ad9bf896098363 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b32e89ffe9d64c5191ad9bf896098363 2023-04-03 05:55:46,297 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b32e89ffe9d64c5191ad9bf896098363, entries=150, sequenceid=57, filesize=11.6 K 2023-04-03 05:55:46,297 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/312d7ba753034b968e50d8da1e67d9ff as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/312d7ba753034b968e50d8da1e67d9ff 2023-04-03 05:55:46,301 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/312d7ba753034b968e50d8da1e67d9ff, entries=150, sequenceid=57, filesize=11.6 K 2023-04-03 05:55:46,302 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=140.89 KB/144270 for 5d777b61781e184e82ec86ca4072f25a in 460ms, sequenceid=57, compaction requested=true 2023-04-03 05:55:46,302 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:46,302 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:46,302 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:46,304 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,304 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,305 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,305 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:46,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6792ed68[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:46,307 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:46,307 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,308 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,308 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:46,308 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:46,308 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:46,308 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,308 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:46,309 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:46,309 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:46,309 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:46,309 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:46,313 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:46,313 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:46,313 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:46,313 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:46,313 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:46,313 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:46,314 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,314 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,315 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:46,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,315 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,316 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,319 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,319 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:46,319 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:46,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:46,319 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,319 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:46,319 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:46,319 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:46,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:46,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:46,319 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:46,320 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:46,320 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,320 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,320 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:46,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:46,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:46,367 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-04-03 05:55:46,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:46,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:46,367 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:46,367 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:46,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:46,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:46,389 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501406382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501406388, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501406389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,395 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501406389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,399 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/c889db32e61e45cca70149c3352ae11d 2023-04-03 05:55:46,420 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:46,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:46,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:46,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:46,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:46,423 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:46,423 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:46,423 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:46,423 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:46,423 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,423 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,424 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,424 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:46,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:46,425 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,425 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:46,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:46,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:46,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:46,426 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:46,426 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:46,426 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:46,426 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:46,426 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:46,427 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,427 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:46,427 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:46,428 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,428 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,428 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,428 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:46,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e10f928[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:46,430 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:46,430 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,432 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,432 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:46,432 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:46,432 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,432 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:46,432 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:46,434 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:46,434 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:46,434 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:46,437 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:46,439 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:46,439 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:46,439 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:46,439 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:46,439 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:46,439 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:46,442 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,442 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,442 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:46,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:46,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:46,442 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,444 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:46,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:46,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:46,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:46,451 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:46,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:46,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:46,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:46,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:46,451 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:46,451 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:46,498 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501406490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,499 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501406494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501406496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501406496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,552 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:46,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:46,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:46,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:46,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:46,555 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:46,555 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:46,555 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:46,555 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:46,557 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,557 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,559 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:46,559 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,559 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,559 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:46,560 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:46,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:46,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:46,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:46,564 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:46,567 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:46,568 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:46,568 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:46,568 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:46,568 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:46,584 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:46,584 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:46,584 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:46,584 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,584 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:46,584 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:46,587 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,588 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,588 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:46,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:46,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cff2b6[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:46,589 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:46,590 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501406582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,592 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:46,592 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:46,592 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:46,592 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:46,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:46,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:46,595 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:46,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:46,601 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:46,601 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:46,601 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:46,601 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:46,601 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:46,601 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:46,608 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,608 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:46,608 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:46,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,611 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,613 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,616 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:46,616 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:46,616 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,616 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,616 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,616 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,616 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,616 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,616 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,616 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:46,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:46,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:46,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:46,617 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:46,617 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:46,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:46,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:46,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501406699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501406702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501406704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:46,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501406705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,717 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:46,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:46,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:46,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:46,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:46,721 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:46,721 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:46,721 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:46,721 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:46,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,722 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,723 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,723 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:46,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:46,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:46,723 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,723 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:46,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,724 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:46,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:46,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:46,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:46,724 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:46,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:46,725 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:46,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:46,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:46,740 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:46,740 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:46,740 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:46,740 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:46,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:46,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:46,740 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:46,741 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:46,742 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:46,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70497fc1[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:46,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:46,744 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,746 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,746 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:46,746 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:46,746 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:46,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:46,748 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:46,748 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:46,748 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:46,748 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:46,751 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:46,751 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:46,751 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:46,751 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:46,751 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:46,751 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:46,754 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,754 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,754 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:46,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,755 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,759 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:46,759 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:46,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:46,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:46,759 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,760 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,760 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:46,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:46,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:46,760 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:46,760 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:46,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,760 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,760 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,760 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,760 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:46,816 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/7d3f8a709c584333b5a40a921eeb92d3 2023-04-03 05:55:46,860 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:46,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:46,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:46,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:46,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:46,872 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:46,872 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:46,872 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:46,872 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:46,873 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,873 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,875 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:46,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:46,875 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,876 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:46,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:46,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:46,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:46,882 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:46,882 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:46,882 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:46,882 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:46,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,884 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:46,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c0350d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:46,886 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:46,886 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,887 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,887 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:46,887 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:46,887 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:46,887 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,887 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:46,888 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,888 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:46,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:46,891 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:46,891 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:46,891 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:46,891 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:46,891 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:46,894 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,894 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,894 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:46,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:46,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:46,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:46,894 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:46,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,895 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:46,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:46,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:46,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:46,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:46,898 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:46,898 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:46,900 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,900 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,900 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:46,900 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:46,900 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,900 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:46,900 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:46,901 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:46,901 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:46,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:46,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:46,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:46,998 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:46,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:47,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:47,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:47,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:47,002 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:47,002 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,002 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:47,002 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:47,002 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,002 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,004 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,004 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,004 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,004 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:47,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:47,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:47,005 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:47,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:47,005 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:47,005 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,005 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:47,005 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:47,005 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:47,005 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,005 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:47,005 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:47,006 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:47,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:47,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:47,006 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,007 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,008 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:47,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9904584[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:47,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:47,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,011 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,011 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:47,011 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:47,011 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:47,011 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,011 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:47,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501407003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:47,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,012 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,014 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501407008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:47,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:47,015 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:47,015 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:47,015 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:47,015 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:47,015 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:47,016 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,016 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,016 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,018 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,023 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,023 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,023 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:47,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:47,023 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:47,023 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:47,023 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:47,023 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,023 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,023 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501407014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,024 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,024 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,024 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,024 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501407015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,096 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501407094, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,123 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:47,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:47,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:47,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:47,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:47,127 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:47,127 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,127 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:47,127 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:47,127 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,127 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,128 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,128 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,129 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,129 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,129 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:47,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:47,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:47,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:47,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:47,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:47,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:47,130 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,130 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:47,130 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:47,130 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:47,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:47,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:47,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:47,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,132 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,132 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,132 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:47,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71b25c42[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:47,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:47,134 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,134 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,135 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,135 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:47,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,137 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:47,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:47,139 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:47,139 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:47,139 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:47,139 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:47,140 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:47,141 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,141 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,141 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,142 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,145 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:47,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:47,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:47,146 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:47,146 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:47,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,234 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=94 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/58e3e006dc8445659f30e6548c63ab74 2023-04-03 05:55:47,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/c889db32e61e45cca70149c3352ae11d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/c889db32e61e45cca70149c3352ae11d 2023-04-03 05:55:47,242 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/c889db32e61e45cca70149c3352ae11d, entries=200, sequenceid=94, filesize=13.9 K 2023-04-03 05:55:47,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/7d3f8a709c584333b5a40a921eeb92d3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/7d3f8a709c584333b5a40a921eeb92d3 2023-04-03 05:55:47,246 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:47,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:47,247 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/7d3f8a709c584333b5a40a921eeb92d3, entries=150, sequenceid=94, filesize=11.6 K 2023-04-03 05:55:47,248 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/58e3e006dc8445659f30e6548c63ab74 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/58e3e006dc8445659f30e6548c63ab74 2023-04-03 05:55:47,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:47,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:47,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:47,249 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:47,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:47,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:47,250 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,250 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,251 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,251 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/58e3e006dc8445659f30e6548c63ab74, entries=150, sequenceid=94, filesize=11.6 K 2023-04-03 05:55:47,251 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,251 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,252 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,252 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 5d777b61781e184e82ec86ca4072f25a in 885ms, sequenceid=94, compaction requested=true 2023-04-03 05:55:47,252 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,252 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:55:47,252 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:47,252 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:47,253 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:55:47,253 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:47,253 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:47,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:47,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:47,265 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:47,265 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,265 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:47,265 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49884 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:55:47,265 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:55:47,265 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:47,265 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:47,265 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,265 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/A is initiating minor compaction (all files) 2023-04-03 05:55:47,265 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:47,265 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/A in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,265 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/B is initiating minor compaction (all files) 2023-04-03 05:55:47,265 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-03 05:55:47,265 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9cea537836f14288a89cbd2207ee44d0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/3394f69cdd6946c38e4b2998daf786d8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/730488298f7c4a1ab323a7f83ce2bd01, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/c889db32e61e45cca70149c3352ae11d] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=48.7 K 2023-04-03 05:55:47,265 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/B in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,266 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/a2fb9ad8a04449f5a43f02f3bdee0d19, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b365c783d3fe4702b5254b51b6bbf9ee, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b32e89ffe9d64c5191ad9bf896098363, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/7d3f8a709c584333b5a40a921eeb92d3] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=46.4 K 2023-04-03 05:55:47,266 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:47,266 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 9cea537836f14288a89cbd2207ee44d0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680501344728 2023-04-03 05:55:47,266 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:47,266 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a2fb9ad8a04449f5a43f02f3bdee0d19, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680501344728 2023-04-03 05:55:47,266 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:47,266 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b365c783d3fe4702b5254b51b6bbf9ee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680501344773 2023-04-03 05:55:47,266 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:47,266 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 3394f69cdd6946c38e4b2998daf786d8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680501344773 2023-04-03 05:55:47,266 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:47,267 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:47,267 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b32e89ffe9d64c5191ad9bf896098363, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1680501345371 2023-04-03 05:55:47,267 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 730488298f7c4a1ab323a7f83ce2bd01, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1680501345371 2023-04-03 05:55:47,267 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 7d3f8a709c584333b5a40a921eeb92d3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680501346027 2023-04-03 05:55:47,267 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting c889db32e61e45cca70149c3352ae11d, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680501346027 2023-04-03 05:55:47,282 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#A#compaction#440 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:47,284 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#B#compaction#441 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:47,296 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/db8afb9daca94fd38e41addbe5f54d41 2023-04-03 05:55:47,311 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/d8651c68a5ca4cef8afee35bbcf914ac 2023-04-03 05:55:47,317 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/26b2f51e902a4cbc8088d53f06f96bf9 2023-04-03 05:55:47,320 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/db8afb9daca94fd38e41addbe5f54d41 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/db8afb9daca94fd38e41addbe5f54d41 2023-04-03 05:55:47,323 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/db8afb9daca94fd38e41addbe5f54d41, entries=150, sequenceid=102, filesize=11.6 K 2023-04-03 05:55:47,324 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/d8651c68a5ca4cef8afee35bbcf914ac as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/d8651c68a5ca4cef8afee35bbcf914ac 2023-04-03 05:55:47,326 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/d8651c68a5ca4cef8afee35bbcf914ac, entries=150, sequenceid=102, filesize=11.6 K 2023-04-03 05:55:47,327 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/26b2f51e902a4cbc8088d53f06f96bf9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/26b2f51e902a4cbc8088d53f06f96bf9 2023-04-03 05:55:47,330 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/26b2f51e902a4cbc8088d53f06f96bf9, entries=150, sequenceid=102, filesize=11.6 K 2023-04-03 05:55:47,330 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 5d777b61781e184e82ec86ca4072f25a in 65ms, sequenceid=102, compaction requested=true 2023-04-03 05:55:47,331 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:47,331 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:47,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:47,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:47,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:47,333 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,333 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:47,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:47,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4abe92c2[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:47,335 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:47,335 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,336 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,336 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:47,336 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:47,336 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:47,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:47,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:47,338 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,338 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:47,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:47,341 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:47,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:47,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:47,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:47,341 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:47,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,344 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,345 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,354 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:47,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:47,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,354 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,355 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:47,355 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:47,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,455 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:47,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:47,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:47,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:47,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:47,458 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:47,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:47,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:47,459 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,459 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,460 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,460 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,460 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:47,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:47,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:47,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:47,462 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:47,462 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,462 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:47,462 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:47,462 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:47,462 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,464 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,464 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:47,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:47,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b57bec5[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:47,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:47,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,467 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,467 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:47,468 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,468 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:47,468 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:47,468 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:47,473 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:47,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:47,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:47,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:47,476 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:47,476 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:47,476 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:47,476 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:47,476 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:47,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,480 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,484 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:47,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:47,484 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,484 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:47,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:47,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,484 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:47,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,496 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 05:55:47,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:47,548 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:55:47,548 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:47,548 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:47,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:47,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:47,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:47,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:47,554 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/ba3970e45d434237979ea44424c856ed 2023-04-03 05:55:47,584 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:47,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:47,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:47,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:47,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:47,587 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:47,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:47,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:47,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,591 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,591 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:47,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:47,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:47,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:47,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:47,593 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:47,593 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,593 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:47,593 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:47,594 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:47,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:47,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:47,594 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:47,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501347572, max timestamp=1680501347596Num uniques 859; 2023-04-03 05:55:47,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:55:47,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501347572, max timestamp=1680501347596Num uniques 859; 2023-04-03 05:55:47,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:55:47,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501347572, max timestamp=1680501347596Num uniques 859; 2023-04-03 05:55:47,598 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:55:47,601 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,601 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:47,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40606e4e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:47,605 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:47,605 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,607 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:47,607 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:47,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:47,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:47,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,613 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:47,613 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:47,613 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:47,613 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:47,613 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:47,613 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:47,613 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:47,614 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,614 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,615 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,615 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,616 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:47,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:47,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,620 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:47,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:47,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,620 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:47,620 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,621 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,621 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501407627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501407634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501407635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501407635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,698 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/44a417f5225c4775b79f0ea995828736 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/44a417f5225c4775b79f0ea995828736 2023-04-03 05:55:47,702 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/A of 5d777b61781e184e82ec86ca4072f25a into 44a417f5225c4775b79f0ea995828736(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:47,702 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:47,702 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/A, priority=12, startTime=1680501347252; duration=0sec 2023-04-03 05:55:47,702 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:47,702 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:55:47,703 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:55:47,703 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/C is initiating minor compaction (all files) 2023-04-03 05:55:47,703 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/C in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,703 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/aa7f32cf79de4496af21c86790e76bc5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/2e65cae376d84ffcb2ab8f8e471b1ed8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/312d7ba753034b968e50d8da1e67d9ff, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/58e3e006dc8445659f30e6548c63ab74, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/26b2f51e902a4cbc8088d53f06f96bf9] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=58.0 K 2023-04-03 05:55:47,704 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting aa7f32cf79de4496af21c86790e76bc5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=12, earliestPutTs=1680501344728 2023-04-03 05:55:47,704 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 2e65cae376d84ffcb2ab8f8e471b1ed8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680501344773 2023-04-03 05:55:47,704 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 312d7ba753034b968e50d8da1e67d9ff, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1680501345371 2023-04-03 05:55:47,704 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 58e3e006dc8445659f30e6548c63ab74, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680501346027 2023-04-03 05:55:47,705 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 26b2f51e902a4cbc8088d53f06f96bf9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680501346377 2023-04-03 05:55:47,709 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/1b0bbc6ab44e4d14b73186a83a2eb99e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1b0bbc6ab44e4d14b73186a83a2eb99e 2023-04-03 05:55:47,712 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/B of 5d777b61781e184e82ec86ca4072f25a into 1b0bbc6ab44e4d14b73186a83a2eb99e(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:47,712 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:47,713 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/B, priority=12, startTime=1680501347253; duration=0sec 2023-04-03 05:55:47,713 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:47,714 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#C#compaction#446 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:47,721 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:47,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:47,722 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/e58cacc76b0b4a49b604efe904b79ea9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/e58cacc76b0b4a49b604efe904b79ea9 2023-04-03 05:55:47,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:47,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:47,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:47,724 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:47,724 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,724 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:47,724 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:47,725 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,725 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,726 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,726 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:47,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,729 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/C of 5d777b61781e184e82ec86ca4072f25a into e58cacc76b0b4a49b604efe904b79ea9(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:47,729 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:47,729 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/C, priority=11, startTime=1680501347253; duration=0sec 2023-04-03 05:55:47,729 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,732 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:47,732 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:47,732 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:47,732 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:47,732 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,732 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:47,736 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,736 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:47,736 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:47,736 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:47,736 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:47,738 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,738 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,738 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,738 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:47,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:47,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23d36a4c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:47,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:47,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,742 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501407737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,743 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,743 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:47,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:47,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:47,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501407737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:47,745 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,745 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:47,745 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,745 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:47,748 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:47,748 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:47,748 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:47,748 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:47,749 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:47,749 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:47,750 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,750 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,750 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,751 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501407744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501407744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,753 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,755 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,755 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,756 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,756 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,756 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,755 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,756 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:47,756 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,756 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:47,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:47,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,756 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,756 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:47,757 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:47,857 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:47,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:47,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:47,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:47,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:47,860 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:47,861 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,861 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:47,861 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:47,861 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,861 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,862 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,863 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,863 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,864 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:47,865 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:47,865 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:47,865 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,865 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:47,865 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:47,865 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:47,865 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:47,868 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,869 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,869 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:47,870 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dc5c476[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:47,870 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:47,870 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,871 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,871 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:47,871 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:47,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:47,872 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,872 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,873 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:47,873 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:47,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:47,878 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:47,878 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:47,878 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:47,878 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:47,878 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:47,880 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,880 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,880 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,881 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,882 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:47,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:47,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,885 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:47,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,885 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:47,885 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:47,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:47,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:47,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,946 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501407944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,949 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501407945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501407953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,955 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:47,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501407953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/133978a2bc104b20abf28566e8625619 2023-04-03 05:55:47,985 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:47,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:47,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:47,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:47,989 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:47,989 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,989 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:47,989 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:47,989 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,989 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,990 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,990 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:47,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:47,990 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,990 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:47,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:47,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:47,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:47,992 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,992 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:47,992 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:47,992 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:47,996 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:47,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b92f09d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:47,997 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:47,997 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,000 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,000 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:48,000 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:48,000 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:48,000 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:48,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,001 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:48,001 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:48,001 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:48,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:48,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:48,003 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:48,004 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:48,004 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:48,004 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:48,004 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:48,005 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,005 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,005 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:48,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:48,005 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,007 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,009 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:48,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:48,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:48,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:48,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:48,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:48,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:48,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:48,009 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:48,010 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:48,110 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:48,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:48,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501408107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:48,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:48,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:48,116 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:48,116 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:48,116 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:48,116 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:48,116 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,117 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,119 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:48,119 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:48,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:48,120 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,120 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:48,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:48,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,121 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:48,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:48,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:48,128 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:48,128 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:48,128 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:48,128 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:48,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:48,130 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:48,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:48,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:48,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2843e1bf[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:48,132 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:48,132 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,133 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,133 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:48,133 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:48,133 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:48,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:48,135 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:48,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:48,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:48,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:48,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:48,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:48,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:48,138 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:48,138 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:48,138 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:48,138 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:48,138 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:48,147 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,148 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:48,148 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,148 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:48,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,149 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:48,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:48,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:48,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:48,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:48,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,152 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:48,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:48,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:48,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:48,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:48,153 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:48,153 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:48,253 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:48,253 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501408249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:48,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:48,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:48,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:48,256 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:48,256 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:48,256 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:48,257 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:48,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501408252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,258 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,258 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,259 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:48,259 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:48,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:48,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,260 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,260 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:48,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,260 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:48,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:48,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:48,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:48,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:48,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501408256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:48,261 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501408258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,262 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:48,262 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:48,262 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:48,262 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:48,262 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:48,263 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:48,264 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,264 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:48,264 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,264 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:48,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:48,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@107a6ff1[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:48,266 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:48,266 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,268 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,268 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:48,268 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:48,268 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:48,269 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:48,270 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:48,270 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:48,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:48,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:48,270 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:48,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:48,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:48,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:48,270 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:48,272 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,272 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:48,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,272 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:48,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:48,273 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:48,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:48,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:48,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:48,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:48,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:48,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:48,281 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:48,281 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:48,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:48,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:48,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,282 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:48,282 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:48,282 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,373 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=113 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/1f8804eb0e6a49c69208812d3ea7eb97 2023-04-03 05:55:48,376 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/ba3970e45d434237979ea44424c856ed as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ba3970e45d434237979ea44424c856ed 2023-04-03 05:55:48,380 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ba3970e45d434237979ea44424c856ed, entries=200, sequenceid=113, filesize=13.9 K 2023-04-03 05:55:48,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/133978a2bc104b20abf28566e8625619 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/133978a2bc104b20abf28566e8625619 2023-04-03 05:55:48,382 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:48,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:48,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/133978a2bc104b20abf28566e8625619, entries=150, sequenceid=113, filesize=11.6 K 2023-04-03 05:55:48,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:48,384 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/1f8804eb0e6a49c69208812d3ea7eb97 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/1f8804eb0e6a49c69208812d3ea7eb97 2023-04-03 05:55:48,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:48,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:48,385 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:48,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:48,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:48,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:48,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,387 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/1f8804eb0e6a49c69208812d3ea7eb97, entries=150, sequenceid=113, filesize=11.6 K 2023-04-03 05:55:48,388 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:48,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:48,388 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,388 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5d777b61781e184e82ec86ca4072f25a in 840ms, sequenceid=113, compaction requested=true 2023-04-03 05:55:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,388 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,388 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:48,388 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:48,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:48,389 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:48,388 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:48,389 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:48,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:48,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:48,389 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:48,389 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:48,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:48,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:48,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:48,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:48,390 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:48,390 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:48,390 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:48,390 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/A is initiating minor compaction (all files) 2023-04-03 05:55:48,390 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/B is initiating minor compaction (all files) 2023-04-03 05:55:48,390 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:48,390 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/B in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:48,390 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/A in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:48,390 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1b0bbc6ab44e4d14b73186a83a2eb99e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/d8651c68a5ca4cef8afee35bbcf914ac, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/133978a2bc104b20abf28566e8625619] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=35.0 K 2023-04-03 05:55:48,390 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:48,390 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/44a417f5225c4775b79f0ea995828736, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/db8afb9daca94fd38e41addbe5f54d41, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ba3970e45d434237979ea44424c856ed] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=37.2 K 2023-04-03 05:55:48,390 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-03 05:55:48,390 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1b0bbc6ab44e4d14b73186a83a2eb99e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680501346027 2023-04-03 05:55:48,390 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:48,390 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 44a417f5225c4775b79f0ea995828736, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=94, earliestPutTs=1680501346027 2023-04-03 05:55:48,390 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:48,391 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:48,391 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting d8651c68a5ca4cef8afee35bbcf914ac, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680501346377 2023-04-03 05:55:48,391 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:48,391 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:48,391 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 133978a2bc104b20abf28566e8625619, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680501347531 2023-04-03 05:55:48,391 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting db8afb9daca94fd38e41addbe5f54d41, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680501346377 2023-04-03 05:55:48,391 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:48,391 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting ba3970e45d434237979ea44424c856ed, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680501347524 2023-04-03 05:55:48,397 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#A#compaction#450 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:48,397 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#B#compaction#449 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:48,410 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/ce789affcbd34fe3aaf0b0d0d3736260 2023-04-03 05:55:48,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:48,763 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:48,774 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501408767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,777 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501408771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501408773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,779 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501408774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,811 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/cba42305e1714bb3b454ef46803640b0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/cba42305e1714bb3b454ef46803640b0 2023-04-03 05:55:48,813 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/581e1388996f466fb317329cd215c32b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/581e1388996f466fb317329cd215c32b 2023-04-03 05:55:48,816 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/B of 5d777b61781e184e82ec86ca4072f25a into cba42305e1714bb3b454ef46803640b0(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:48,816 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:48,816 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/B, priority=13, startTime=1680501348389; duration=0sec 2023-04-03 05:55:48,816 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:48,816 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-03 05:55:48,817 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 05:55:48,817 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 05:55:48,817 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. because compaction request was cancelled 2023-04-03 05:55:48,819 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/db644e96d71e4c42a118a369cf034d3d 2023-04-03 05:55:48,819 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/A of 5d777b61781e184e82ec86ca4072f25a into 581e1388996f466fb317329cd215c32b(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:48,819 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:48,819 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/A, priority=13, startTime=1680501348388; duration=0sec 2023-04-03 05:55:48,819 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:48,878 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501408875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,883 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501408878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501408880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:48,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:48,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501408880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,083 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501409081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501409085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501409086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501409087, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,228 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=152 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/cec18d6dbd6a4683ab29db90ed4457ba 2023-04-03 05:55:49,235 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/ce789affcbd34fe3aaf0b0d0d3736260 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ce789affcbd34fe3aaf0b0d0d3736260 2023-04-03 05:55:49,238 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ce789affcbd34fe3aaf0b0d0d3736260, entries=150, sequenceid=152, filesize=11.8 K 2023-04-03 05:55:49,239 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/db644e96d71e4c42a118a369cf034d3d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/db644e96d71e4c42a118a369cf034d3d 2023-04-03 05:55:49,242 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/db644e96d71e4c42a118a369cf034d3d, entries=150, sequenceid=152, filesize=11.8 K 2023-04-03 05:55:49,243 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/cec18d6dbd6a4683ab29db90ed4457ba as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cec18d6dbd6a4683ab29db90ed4457ba 2023-04-03 05:55:49,257 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cec18d6dbd6a4683ab29db90ed4457ba, entries=150, sequenceid=152, filesize=11.8 K 2023-04-03 05:55:49,258 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 5d777b61781e184e82ec86ca4072f25a in 868ms, sequenceid=152, compaction requested=true 2023-04-03 05:55:49,258 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:49,258 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:49,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:49,260 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:49,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:49,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:49,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23afeffc[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:49,262 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:49,262 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,263 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:49,264 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:49,264 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,264 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:49,269 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,269 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:49,269 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:49,269 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:49,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:49,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:49,271 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:49,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:49,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:49,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:49,271 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:49,273 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,273 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,273 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:49,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:49,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:49,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:49,276 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:49,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:49,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:49,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:49,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:49,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:49,279 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:49,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:49,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:49,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:49,279 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:49,279 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:49,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:49,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:49,379 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:49,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:49,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:49,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:49,391 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 05:55:49,391 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:49,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:49,391 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:49,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:49,391 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:49,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:49,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:49,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:49,394 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:49,394 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:49,394 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:49,394 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:49,395 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,395 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:49,396 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,396 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:49,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:49,396 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:49,396 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:49,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:49,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:49,411 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:49,411 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:49,411 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:49,411 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:49,411 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:49,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:49,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:49,421 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:49,421 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:49,421 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:49,421 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:49,422 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:49,422 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:49,422 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:49,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:49,446 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,446 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,447 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:49,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:49,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fc5f50c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:49,449 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:49,449 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,450 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,450 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:49,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:49,451 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:49,452 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:49,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:49,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:49,456 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:49,456 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:49,456 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:49,456 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:49,456 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:49,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:49,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:49,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:49,462 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:49,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:49,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,464 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/9e3752eb5d60451cb829c17378ed6920 2023-04-03 05:55:49,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:49,465 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501349431, max timestamp=1680501349463Num uniques 859; 2023-04-03 05:55:49,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:55:49,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501349431, max timestamp=1680501349463Num uniques 859; 2023-04-03 05:55:49,466 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:55:49,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:49,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501349431, max timestamp=1680501349463Num uniques 859; 2023-04-03 05:55:49,468 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:55:49,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,475 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:49,475 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:49,475 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:49,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:49,475 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:49,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:49,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:49,475 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:49,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:49,476 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:49,476 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:49,476 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:49,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:49,476 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,476 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,476 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:49,476 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:49,491 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/a79aae389e6243468386fe1ff0174b73 2023-04-03 05:55:49,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501409508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501409509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501409517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,528 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/23978a2bbe8f44d5b273c9d3fca79605 2023-04-03 05:55:49,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501409521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,538 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/9e3752eb5d60451cb829c17378ed6920 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9e3752eb5d60451cb829c17378ed6920 2023-04-03 05:55:49,543 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9e3752eb5d60451cb829c17378ed6920, entries=200, sequenceid=165, filesize=14.1 K 2023-04-03 05:55:49,544 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/a79aae389e6243468386fe1ff0174b73 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/a79aae389e6243468386fe1ff0174b73 2023-04-03 05:55:49,574 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/a79aae389e6243468386fe1ff0174b73, entries=150, sequenceid=165, filesize=11.8 K 2023-04-03 05:55:49,575 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/23978a2bbe8f44d5b273c9d3fca79605 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/23978a2bbe8f44d5b273c9d3fca79605 2023-04-03 05:55:49,576 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:49,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:49,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:49,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/23978a2bbe8f44d5b273c9d3fca79605, entries=150, sequenceid=165, filesize=11.8 K 2023-04-03 05:55:49,580 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5d777b61781e184e82ec86ca4072f25a in 189ms, sequenceid=165, compaction requested=true 2023-04-03 05:55:49,580 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:49,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:49,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:49,580 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:49,580 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 05:55:49,580 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:55:49,581 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38588 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:49,581 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/A is initiating minor compaction (all files) 2023-04-03 05:55:49,581 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/A in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:49,582 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/581e1388996f466fb317329cd215c32b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ce789affcbd34fe3aaf0b0d0d3736260, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9e3752eb5d60451cb829c17378ed6920] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=37.7 K 2023-04-03 05:55:49,582 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48016 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:55:49,582 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/C is initiating minor compaction (all files) 2023-04-03 05:55:49,582 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/C in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:49,582 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/e58cacc76b0b4a49b604efe904b79ea9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/1f8804eb0e6a49c69208812d3ea7eb97, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cec18d6dbd6a4683ab29db90ed4457ba, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/23978a2bbe8f44d5b273c9d3fca79605] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=46.9 K 2023-04-03 05:55:49,583 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e58cacc76b0b4a49b604efe904b79ea9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1680501346377 2023-04-03 05:55:49,583 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 581e1388996f466fb317329cd215c32b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680501347531 2023-04-03 05:55:49,583 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1f8804eb0e6a49c69208812d3ea7eb97, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680501347531 2023-04-03 05:55:49,583 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting ce789affcbd34fe3aaf0b0d0d3736260, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1680501347624 2023-04-03 05:55:49,584 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting cec18d6dbd6a4683ab29db90ed4457ba, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1680501347624 2023-04-03 05:55:49,584 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 9e3752eb5d60451cb829c17378ed6920, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680501348766 2023-04-03 05:55:49,584 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 23978a2bbe8f44d5b273c9d3fca79605, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680501348766 2023-04-03 05:55:49,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:49,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:49,585 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:49,585 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:49,585 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:49,585 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:49,586 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,586 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:49,588 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:49,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:49,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:49,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:49,590 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,590 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:49,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:49,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:49,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:49,604 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:49,604 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:49,604 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:49,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:49,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:49,605 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:49,605 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:49,605 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:49,606 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-03 05:55:49,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:49,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:49,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:49,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:49,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:49,606 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:49,607 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#C#compaction#456 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:49,613 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#A#compaction#457 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:49,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:49,632 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:49,638 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/2efaf0a3318549599e9d2766e3c6d3be as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/2efaf0a3318549599e9d2766e3c6d3be 2023-04-03 05:55:49,643 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/A of 5d777b61781e184e82ec86ca4072f25a into 2efaf0a3318549599e9d2766e3c6d3be(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:49,643 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:49,643 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/A, priority=13, startTime=1680501349580; duration=0sec 2023-04-03 05:55:49,644 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:49,644 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/adae205c968b415a9ea5feac4292b17a 2023-04-03 05:55:49,644 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:49,648 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36198 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:49,648 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/B is initiating minor compaction (all files) 2023-04-03 05:55:49,648 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/B in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:49,648 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/cba42305e1714bb3b454ef46803640b0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/db644e96d71e4c42a118a369cf034d3d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/a79aae389e6243468386fe1ff0174b73] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=35.3 K 2023-04-03 05:55:49,652 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting cba42305e1714bb3b454ef46803640b0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=113, earliestPutTs=1680501347531 2023-04-03 05:55:49,653 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting db644e96d71e4c42a118a369cf034d3d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=152, earliestPutTs=1680501347624 2023-04-03 05:55:49,654 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting a79aae389e6243468386fe1ff0174b73, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680501348766 2023-04-03 05:55:49,665 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#B#compaction#460 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:49,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501409677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501409677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,691 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501409677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501409678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501409792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501409792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501409792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:49,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:49,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501409793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501409997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501409997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,003 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501409998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,004 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501409998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,030 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/6953e1f2804e4cff83cd0150a76d8b84 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/6953e1f2804e4cff83cd0150a76d8b84 2023-04-03 05:55:50,033 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/C of 5d777b61781e184e82ec86ca4072f25a into 6953e1f2804e4cff83cd0150a76d8b84(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:50,033 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:50,033 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/C, priority=12, startTime=1680501349580; duration=0sec 2023-04-03 05:55:50,034 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:50,065 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/74d9c4409fb44344948167e5f4291385 2023-04-03 05:55:50,077 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/470c5eed07de4770970e7e68db7b948f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/470c5eed07de4770970e7e68db7b948f 2023-04-03 05:55:50,081 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/B of 5d777b61781e184e82ec86ca4072f25a into 470c5eed07de4770970e7e68db7b948f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:50,081 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:50,081 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/B, priority=13, startTime=1680501349580; duration=0sec 2023-04-03 05:55:50,081 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:50,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501410116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,119 DEBUG [Thread-3286] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:55:50,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501410304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501410304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,309 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501410305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,310 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501410307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,476 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/174a5951d4594d33a46b30a624645466 2023-04-03 05:55:50,480 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/adae205c968b415a9ea5feac4292b17a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/adae205c968b415a9ea5feac4292b17a 2023-04-03 05:55:50,484 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/adae205c968b415a9ea5feac4292b17a, entries=150, sequenceid=199, filesize=11.8 K 2023-04-03 05:55:50,485 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/74d9c4409fb44344948167e5f4291385 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/74d9c4409fb44344948167e5f4291385 2023-04-03 05:55:50,490 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/74d9c4409fb44344948167e5f4291385, entries=150, sequenceid=199, filesize=11.8 K 2023-04-03 05:55:50,490 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/174a5951d4594d33a46b30a624645466 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/174a5951d4594d33a46b30a624645466 2023-04-03 05:55:50,493 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/174a5951d4594d33a46b30a624645466, entries=150, sequenceid=199, filesize=11.8 K 2023-04-03 05:55:50,494 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 5d777b61781e184e82ec86ca4072f25a in 888ms, sequenceid=199, compaction requested=false 2023-04-03 05:55:50,494 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:50,494 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:50,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:50,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:50,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:50,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:50,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:50,496 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:50,497 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:50,497 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:50,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:50,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:50,498 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68bcb952[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:50,498 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:50,498 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:50,501 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:50,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:50,501 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,501 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:50,501 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:50,502 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,502 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:50,502 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:50,502 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:50,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:50,505 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:50,505 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:50,505 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:50,505 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:50,505 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:50,506 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:50,506 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:50,506 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:50,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:50,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:50,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:50,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:50,510 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:50,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,515 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,517 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:50,517 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:50,517 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:50,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:50,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:50,517 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:50,517 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:50,517 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:50,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:50,517 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:50,517 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:50,518 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,518 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:50,518 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:50,518 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:50,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:50,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:50,618 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:50,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:50,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:50,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:50,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:50,621 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:50,621 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:50,621 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:50,621 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:50,621 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:50,621 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:50,625 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,625 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:50,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:50,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:50,626 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:50,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:50,627 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:50,627 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:50,627 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:50,627 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:50,627 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:50,627 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 05:55:50,627 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:50,627 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:50,627 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:50,628 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:50,628 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:50,628 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:50,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:50,814 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:50,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501350831, max timestamp=1680501350851Num uniques 859; 2023-04-03 05:55:50,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:55:50,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501350831, max timestamp=1680501350851Num uniques 859; 2023-04-03 05:55:50,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:55:50,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501350831, max timestamp=1680501350851Num uniques 859; 2023-04-03 05:55:50,853 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:55:50,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501410881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,886 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501410882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501410883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501410885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501410985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501410987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501410987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:50,997 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:50,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501410995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,034 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/8187dd7671014c98a434e984d259f8cd 2023-04-03 05:55:51,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:51,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501411191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:51,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501411192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:51,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501411192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:51,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501411198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,443 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/741c07e32db046c1935b667a7b7e9e21 2023-04-03 05:55:51,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:51,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501411497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:51,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501411499, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:51,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501411500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:51,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501411505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,851 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/00c75a671776409fa4d336dfcf222768 2023-04-03 05:55:51,855 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/8187dd7671014c98a434e984d259f8cd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8187dd7671014c98a434e984d259f8cd 2023-04-03 05:55:51,858 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8187dd7671014c98a434e984d259f8cd, entries=150, sequenceid=213, filesize=11.8 K 2023-04-03 05:55:51,859 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/741c07e32db046c1935b667a7b7e9e21 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/741c07e32db046c1935b667a7b7e9e21 2023-04-03 05:55:51,862 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/741c07e32db046c1935b667a7b7e9e21, entries=150, sequenceid=213, filesize=11.8 K 2023-04-03 05:55:51,862 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/00c75a671776409fa4d336dfcf222768 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/00c75a671776409fa4d336dfcf222768 2023-04-03 05:55:51,865 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/00c75a671776409fa4d336dfcf222768, entries=150, sequenceid=213, filesize=11.8 K 2023-04-03 05:55:51,866 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5d777b61781e184e82ec86ca4072f25a in 1239ms, sequenceid=213, compaction requested=true 2023-04-03 05:55:51,866 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:51,866 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:51,866 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:51,867 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,867 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:51,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:51,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:51,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:51,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:51,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bfa4653[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:51,869 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:51,869 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:51,870 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:51,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:51,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:51,870 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:51,870 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:51,870 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:51,870 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,871 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:51,871 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,871 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:51,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:51,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:51,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:51,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:51,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:51,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:51,879 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:51,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:51,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:51,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:51,879 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:51,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:51,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:51,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:51,885 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:51,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:51,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:51,886 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:51,887 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,889 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:51,889 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:51,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:51,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:51,889 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:51,889 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:51,889 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:51,890 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:51,890 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:51,890 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:51,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:51,890 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,890 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:51,890 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:51,890 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:51,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:51,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:51,990 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:51,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:51,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:51,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:51,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:51,993 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:51,993 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:51,993 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:51,993 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:51,994 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:51,994 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:51,995 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,995 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:51,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:51,995 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:51,995 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:51,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:51,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:51,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:51,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:51,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:51,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:51,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:51,996 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:51,996 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:51,996 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-03 05:55:51,997 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:51,997 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:51,997 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:51,997 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:51,997 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:51,997 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:52,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:52,007 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:52,025 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501412016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501412017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501412025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501412026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501412126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501412128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501412134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501412135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,336 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501412333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,337 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501412334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34586 deadline: 1680501412341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,344 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501412342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,404 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/9b4cd55b70c54143b8868061b42d3652 2023-04-03 05:55:52,430 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/2d4e906f8ffe41c8a0b6c1f36a24edcc 2023-04-03 05:55:52,460 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=247 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/0232b6e8d9ea4b7eacfd6a67f3c1cceb 2023-04-03 05:55:52,465 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/9b4cd55b70c54143b8868061b42d3652 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9b4cd55b70c54143b8868061b42d3652 2023-04-03 05:55:52,469 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9b4cd55b70c54143b8868061b42d3652, entries=150, sequenceid=247, filesize=11.8 K 2023-04-03 05:55:52,470 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/2d4e906f8ffe41c8a0b6c1f36a24edcc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/2d4e906f8ffe41c8a0b6c1f36a24edcc 2023-04-03 05:55:52,475 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/2d4e906f8ffe41c8a0b6c1f36a24edcc, entries=150, sequenceid=247, filesize=11.8 K 2023-04-03 05:55:52,475 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/0232b6e8d9ea4b7eacfd6a67f3c1cceb as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/0232b6e8d9ea4b7eacfd6a67f3c1cceb 2023-04-03 05:55:52,479 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/0232b6e8d9ea4b7eacfd6a67f3c1cceb, entries=150, sequenceid=247, filesize=11.8 K 2023-04-03 05:55:52,480 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 5d777b61781e184e82ec86ca4072f25a in 484ms, sequenceid=247, compaction requested=true 2023-04-03 05:55:52,480 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:52,480 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:52,480 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:52,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:52,482 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:52,482 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:52,482 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:52,482 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:52,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:52,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:52,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:52,484 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77f68cdf[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:52,484 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:52,484 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:52,486 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:52,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:52,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:52,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:52,487 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:52,487 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,487 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:52,488 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:52,488 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:52,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:52,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:52,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:52,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:52,491 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:52,491 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:52,491 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:52,491 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:52,491 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:52,493 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:52,493 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:52,493 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:52,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:52,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:52,495 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:52,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:52,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:52,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:52,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,497 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:52,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:52,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:52,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:52,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:52,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:52,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:52,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:52,500 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:52,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:52,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:52,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:52,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:52,500 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:52,501 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:52,501 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:52,501 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:52,601 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:52,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:52,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:52,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:52,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:52,604 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:52,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:52,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:52,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:52,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:52,604 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:52,606 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:52,606 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:52,607 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,607 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:52,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:52,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:52,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:52,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:52,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:52,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:52,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:52,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:52,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:52,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:52,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:52,608 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:52,608 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:52,608 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 05:55:52,609 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:52,609 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:52,609 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:52,609 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:52,609 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:52,609 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:52,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:52,643 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:52,683 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501412672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,683 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680501352663, max timestamp=1680501352681Num uniques 859; 2023-04-03 05:55:52,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680501352663, max timestamp=1680501352681Num uniques 859; 2023-04-03 05:55:52,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:55:52,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:55:52,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680501352663, max timestamp=1680501352681Num uniques 859; 2023-04-03 05:55:52,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501412673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:55:52,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501412752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501412757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501412784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501412785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501412861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,866 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501412862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501412991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:52,998 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:52,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501412992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,020 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/bec0c43acb6f42f09a6f14fb53e57e08 2023-04-03 05:55:53,034 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/0b8007e30fec4dcd89145972838ce08d 2023-04-03 05:55:53,048 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/eefef8ca7d4649a39234cfd435e20c7f 2023-04-03 05:55:53,051 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/bec0c43acb6f42f09a6f14fb53e57e08 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/bec0c43acb6f42f09a6f14fb53e57e08 2023-04-03 05:55:53,054 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/bec0c43acb6f42f09a6f14fb53e57e08, entries=150, sequenceid=258, filesize=11.8 K 2023-04-03 05:55:53,055 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/0b8007e30fec4dcd89145972838ce08d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/0b8007e30fec4dcd89145972838ce08d 2023-04-03 05:55:53,057 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/0b8007e30fec4dcd89145972838ce08d, entries=150, sequenceid=258, filesize=11.8 K 2023-04-03 05:55:53,058 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/eefef8ca7d4649a39234cfd435e20c7f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/eefef8ca7d4649a39234cfd435e20c7f 2023-04-03 05:55:53,064 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/eefef8ca7d4649a39234cfd435e20c7f, entries=150, sequenceid=258, filesize=11.8 K 2023-04-03 05:55:53,065 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 5d777b61781e184e82ec86ca4072f25a in 457ms, sequenceid=258, compaction requested=true 2023-04-03 05:55:53,065 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:53,065 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:53,065 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:53,065 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:53,065 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,065 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:53,065 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:53,067 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,067 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:53,067 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,067 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:53,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:53,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:53,070 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76b7ab8[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:53,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:53,070 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:53,071 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-03 05:55:53,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,071 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:53,071 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:53,071 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:53,072 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,072 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:53,072 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:53,072 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:53,072 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:53,072 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:53,072 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:53,072 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:53,073 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:53,073 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:53,074 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:53,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:53,076 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:53,076 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:53,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:53,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:53,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:53,076 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:53,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:53,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:53,080 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,081 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/eb5e22d6028c4053a48f7c0be03884c9 2023-04-03 05:55:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:53,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,082 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:53,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:53,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:53,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:53,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:53,085 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:53,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:53,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:53,086 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:53,086 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:53,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:53,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:53,105 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/572c354bb0574d8482f37d9532886e38 2023-04-03 05:55:53,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501413106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,123 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501413107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,186 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:53,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:53,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:53,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:53,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:53,189 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:53,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:53,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:53,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:53,190 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,190 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,191 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:53,191 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,191 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,191 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:53,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:53,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:53,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:53,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:53,200 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:53,200 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:53,200 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:53,200 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:53,200 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:53,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,202 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:53,202 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:53,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:53,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:53,204 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a10352e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:53,204 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:53,204 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,208 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:53,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:53,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:53,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,209 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:53,209 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,209 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:53,210 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:53,210 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:53,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:53,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:53,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:53,212 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:53,212 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:53,212 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:53,212 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:53,213 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:53,214 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,214 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:53,214 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:53,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:53,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:53,215 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,217 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:53,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,223 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:53,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:53,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:53,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:53,223 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:53,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:53,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:53,223 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:53,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,223 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:53,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:53,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,224 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:53,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:53,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34662 deadline: 1680501413224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,233 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501413224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501413300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501413301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,324 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:53,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:53,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:53,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:53,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:53,337 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:53,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:53,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:53,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:53,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,339 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:53,339 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:53,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:53,339 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,340 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,340 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:53,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,340 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:53,344 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:53,344 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:53,344 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:53,344 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:53,344 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:53,348 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:53,348 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:53,348 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:53,348 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:53,348 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:53,350 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,350 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:53,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:53,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:53,351 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,351 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:53,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:53,353 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73a132e1[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:53,353 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:53,353 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:53,355 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,355 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:53,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:53,357 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:53,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:53,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:53,360 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:53,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:53,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:53,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:53,360 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:53,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:53,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:53,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:53,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:53,363 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:53,364 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,366 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,366 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,366 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,366 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:53,366 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:53,366 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:53,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:53,366 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:53,366 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:53,367 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,367 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,367 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:53,367 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:53,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:53,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:53,441 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501413434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,442 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501413435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,467 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:53,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:53,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:53,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:53,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:53,470 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:53,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:53,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:53,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:53,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,470 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,472 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:53,472 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:53,473 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,473 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:53,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:53,474 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:53,474 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:53,474 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:53,474 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:53,474 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:53,474 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:53,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:53,475 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:53,476 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:53,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:53,476 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:53,477 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:53,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30f2c83e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:53,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:53,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:53,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:53,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:53,481 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:53,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:53,481 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:53,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:53,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:53,483 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:53,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:53,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:53,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:53,483 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:53,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:53,485 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:53,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:53,486 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:53,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:53,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:53,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:53,489 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:53,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:53,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:53,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:53,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:53,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,489 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:53,489 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:53,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:53,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:53,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:53,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:53,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/cebf34cff22e4e38b3a545c13e19cff3 2023-04-03 05:55:53,526 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/eb5e22d6028c4053a48f7c0be03884c9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/eb5e22d6028c4053a48f7c0be03884c9 2023-04-03 05:55:53,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/eb5e22d6028c4053a48f7c0be03884c9, entries=250, sequenceid=294, filesize=16.7 K 2023-04-03 05:55:53,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/572c354bb0574d8482f37d9532886e38 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/572c354bb0574d8482f37d9532886e38 2023-04-03 05:55:53,533 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/572c354bb0574d8482f37d9532886e38, entries=150, sequenceid=294, filesize=11.9 K 2023-04-03 05:55:53,535 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/cebf34cff22e4e38b3a545c13e19cff3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cebf34cff22e4e38b3a545c13e19cff3 2023-04-03 05:55:53,537 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cebf34cff22e4e38b3a545c13e19cff3, entries=150, sequenceid=294, filesize=11.9 K 2023-04-03 05:55:53,538 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=46.96 KB/48090 for 5d777b61781e184e82ec86ca4072f25a in 467ms, sequenceid=294, compaction requested=true 2023-04-03 05:55:53,538 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:53,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:53,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:53,538 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:53,538 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 05:55:53,538 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 05:55:53,540 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 77638 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 05:55:53,540 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72758 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 05:55:53,540 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/A is initiating minor compaction (all files) 2023-04-03 05:55:53,540 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/B is initiating minor compaction (all files) 2023-04-03 05:55:53,540 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/A in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:53,540 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/B in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:53,540 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/2efaf0a3318549599e9d2766e3c6d3be, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/adae205c968b415a9ea5feac4292b17a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8187dd7671014c98a434e984d259f8cd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9b4cd55b70c54143b8868061b42d3652, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/bec0c43acb6f42f09a6f14fb53e57e08, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/eb5e22d6028c4053a48f7c0be03884c9] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=75.8 K 2023-04-03 05:55:53,540 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/470c5eed07de4770970e7e68db7b948f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/74d9c4409fb44344948167e5f4291385, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/741c07e32db046c1935b667a7b7e9e21, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/2d4e906f8ffe41c8a0b6c1f36a24edcc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/0b8007e30fec4dcd89145972838ce08d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/572c354bb0574d8482f37d9532886e38] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=71.1 K 2023-04-03 05:55:53,540 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 2efaf0a3318549599e9d2766e3c6d3be, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680501348766 2023-04-03 05:55:53,541 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 470c5eed07de4770970e7e68db7b948f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680501348766 2023-04-03 05:55:53,541 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting adae205c968b415a9ea5feac4292b17a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1680501349500 2023-04-03 05:55:53,541 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 74d9c4409fb44344948167e5f4291385, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1680501349500 2023-04-03 05:55:53,541 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8187dd7671014c98a434e984d259f8cd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680501349645 2023-04-03 05:55:53,541 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 741c07e32db046c1935b667a7b7e9e21, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680501349645 2023-04-03 05:55:53,541 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 9b4cd55b70c54143b8868061b42d3652, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680501350881 2023-04-03 05:55:53,541 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 2d4e906f8ffe41c8a0b6c1f36a24edcc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680501350881 2023-04-03 05:55:53,541 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting bec0c43acb6f42f09a6f14fb53e57e08, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680501352016 2023-04-03 05:55:53,542 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0b8007e30fec4dcd89145972838ce08d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680501352016 2023-04-03 05:55:53,542 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting eb5e22d6028c4053a48f7c0be03884c9, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680501352731 2023-04-03 05:55:53,542 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 572c354bb0574d8482f37d9532886e38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680501352731 2023-04-03 05:55:53,550 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#A#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:53,550 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#B#compaction#475 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:53,560 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/c38b1b42fc414137b82612ee0b0dd0eb as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/c38b1b42fc414137b82612ee0b0dd0eb 2023-04-03 05:55:53,563 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/B of 5d777b61781e184e82ec86ca4072f25a into c38b1b42fc414137b82612ee0b0dd0eb(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:53,564 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:53,564 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/B, priority=10, startTime=1680501353538; duration=0sec 2023-04-03 05:55:53,564 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:53,564 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-03 05:55:53,565 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72724 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-03 05:55:53,565 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/C is initiating minor compaction (all files) 2023-04-03 05:55:53,565 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/C in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:53,565 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/6953e1f2804e4cff83cd0150a76d8b84, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/174a5951d4594d33a46b30a624645466, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/00c75a671776409fa4d336dfcf222768, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/0232b6e8d9ea4b7eacfd6a67f3c1cceb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/eefef8ca7d4649a39234cfd435e20c7f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cebf34cff22e4e38b3a545c13e19cff3] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=71.0 K 2023-04-03 05:55:53,565 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6953e1f2804e4cff83cd0150a76d8b84, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680501348766 2023-04-03 05:55:53,566 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 174a5951d4594d33a46b30a624645466, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1680501349500 2023-04-03 05:55:53,566 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 00c75a671776409fa4d336dfcf222768, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680501349645 2023-04-03 05:55:53,566 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0232b6e8d9ea4b7eacfd6a67f3c1cceb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=247, earliestPutTs=1680501350881 2023-04-03 05:55:53,566 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting eefef8ca7d4649a39234cfd435e20c7f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1680501352016 2023-04-03 05:55:53,567 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting cebf34cff22e4e38b3a545c13e19cff3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680501352731 2023-04-03 05:55:53,573 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#C#compaction#476 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:53,579 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/b3e9f04abf274013a5e27c9f5ee7eae4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/b3e9f04abf274013a5e27c9f5ee7eae4 2023-04-03 05:55:53,583 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/C of 5d777b61781e184e82ec86ca4072f25a into b3e9f04abf274013a5e27c9f5ee7eae4(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:53,583 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:53,583 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/C, priority=10, startTime=1680501353538; duration=0sec 2023-04-03 05:55:53,583 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:53,590 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:53,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:53,591 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:53,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:53,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:53,592 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:53,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:53,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:53,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:53,593 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,593 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:53,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:53,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,594 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:53,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:53,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:53,595 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:53,595 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:53,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:53,595 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:53,595 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:53,595 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:53,595 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:55:53,595 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:53,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:53,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:53,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:53,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:53,596 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:53,605 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/e1794f4c05aa418b8b0800186a13b1ee 2023-04-03 05:55:53,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:53,747 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:53,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501353792, max timestamp=1680501353814Num uniques 859; 2023-04-03 05:55:53,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501353792, max timestamp=1680501353814Num uniques 859; 2023-04-03 05:55:53,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501353792, max timestamp=1680501353814Num uniques 859; 2023-04-03 05:55:53,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:55:53,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:55:53,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:55:53,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501413855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501413856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501413863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,870 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501413863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,961 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/579cc046ed894233b255c22e4756dc75 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/579cc046ed894233b255c22e4756dc75 2023-04-03 05:55:53,964 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/A of 5d777b61781e184e82ec86ca4072f25a into 579cc046ed894233b255c22e4756dc75(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:53,964 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:53,964 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/A, priority=10, startTime=1680501353538; duration=0sec 2023-04-03 05:55:53,965 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:53,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501413964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501413964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34608 deadline: 1680501413969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:53,979 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:53,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501413971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,018 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/60d32876aa6a41e4822ebddf04be2324 2023-04-03 05:55:54,155 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501414151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,157 DEBUG [Thread-3286] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8192 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:55:54,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501414170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34614 deadline: 1680501414170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501414172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,185 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501414181, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,426 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/01fcbb95a1c5493fb14750a6ecf7cb27 2023-04-03 05:55:54,430 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/e1794f4c05aa418b8b0800186a13b1ee as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/e1794f4c05aa418b8b0800186a13b1ee 2023-04-03 05:55:54,433 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/e1794f4c05aa418b8b0800186a13b1ee, entries=150, sequenceid=306, filesize=11.9 K 2023-04-03 05:55:54,433 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/60d32876aa6a41e4822ebddf04be2324 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/60d32876aa6a41e4822ebddf04be2324 2023-04-03 05:55:54,437 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/60d32876aa6a41e4822ebddf04be2324, entries=150, sequenceid=306, filesize=11.9 K 2023-04-03 05:55:54,438 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/01fcbb95a1c5493fb14750a6ecf7cb27 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/01fcbb95a1c5493fb14750a6ecf7cb27 2023-04-03 05:55:54,441 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/01fcbb95a1c5493fb14750a6ecf7cb27, entries=150, sequenceid=306, filesize=11.9 K 2023-04-03 05:55:54,442 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5d777b61781e184e82ec86ca4072f25a in 847ms, sequenceid=306, compaction requested=false 2023-04-03 05:55:54,442 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:54,442 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:54,442 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:54,442 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:54,442 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,442 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:54,442 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c90ca22[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:54,446 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:54,446 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,448 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:54,448 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:54,449 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,449 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:54,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:54,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:54,452 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:54,452 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:54,452 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:54,452 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:54,452 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:54,453 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,453 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,454 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:54,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:54,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,455 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:54,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,461 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:54,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:54,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:54,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:54,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:54,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:54,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:54,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:54,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,461 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:54,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,462 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:54,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,462 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:54,480 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-03 05:55:54,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680501353819, max timestamp=1680501354479Num uniques 1068; 2023-04-03 05:55:54,481 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-03 05:55:54,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680501353819, max timestamp=1680501354479Num uniques 1068; 2023-04-03 05:55:54,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-03 05:55:54,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680501353819, max timestamp=1680501354479Num uniques 1068; 2023-04-03 05:55:54,482 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=9978, suffix heap size=139040,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=218 2023-04-03 05:55:54,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:54,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:54,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:54,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:54,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:54,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:54,490 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/b52723a177e64c7c94755d512c9e27c2 2023-04-03 05:55:54,498 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/25be44160438474782ad1131dc4f64fb 2023-04-03 05:55:54,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501354492, max timestamp=1680501354507Num uniques 859; 2023-04-03 05:55:54,508 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:55:54,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501354492, max timestamp=1680501354507Num uniques 859; 2023-04-03 05:55:54,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:55:54,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501354492, max timestamp=1680501354507Num uniques 859; 2023-04-03 05:55:54,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:55:54,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=345 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/41a5c3d05e58452b9ce53cc5ccb964df 2023-04-03 05:55:54,518 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/b52723a177e64c7c94755d512c9e27c2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/b52723a177e64c7c94755d512c9e27c2 2023-04-03 05:55:54,522 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/b52723a177e64c7c94755d512c9e27c2, entries=200, sequenceid=345, filesize=14.3 K 2023-04-03 05:55:54,523 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/25be44160438474782ad1131dc4f64fb as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/25be44160438474782ad1131dc4f64fb 2023-04-03 05:55:54,527 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/25be44160438474782ad1131dc4f64fb, entries=150, sequenceid=345, filesize=11.9 K 2023-04-03 05:55:54,528 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/41a5c3d05e58452b9ce53cc5ccb964df as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/41a5c3d05e58452b9ce53cc5ccb964df 2023-04-03 05:55:54,531 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/41a5c3d05e58452b9ce53cc5ccb964df, entries=150, sequenceid=345, filesize=11.9 K 2023-04-03 05:55:54,531 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~80.58 KB/82512, currentSize=87.22 KB/89310 for 5d777b61781e184e82ec86ca4072f25a in 51ms, sequenceid=345, compaction requested=true 2023-04-03 05:55:54,531 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:54,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:54,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:54,532 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:54,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:54,532 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:54,532 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:54,532 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=93.93 KB heapSize=256.08 KB 2023-04-03 05:55:54,533 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:54,533 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:54,533 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:54,533 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:54,533 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:54,533 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:54,533 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39544 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:54,533 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:54,533 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/A is initiating minor compaction (all files) 2023-04-03 05:55:54,533 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/B is initiating minor compaction (all files) 2023-04-03 05:55:54,533 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/A in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:54,533 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/B in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:54,533 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/579cc046ed894233b255c22e4756dc75, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/e1794f4c05aa418b8b0800186a13b1ee, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/b52723a177e64c7c94755d512c9e27c2] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=38.6 K 2023-04-03 05:55:54,533 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/c38b1b42fc414137b82612ee0b0dd0eb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/60d32876aa6a41e4822ebddf04be2324, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/25be44160438474782ad1131dc4f64fb] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=36.2 K 2023-04-03 05:55:54,534 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 579cc046ed894233b255c22e4756dc75, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680501352731 2023-04-03 05:55:54,535 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting c38b1b42fc414137b82612ee0b0dd0eb, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680501352731 2023-04-03 05:55:54,535 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e1794f4c05aa418b8b0800186a13b1ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680501353088 2023-04-03 05:55:54,535 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting b52723a177e64c7c94755d512c9e27c2, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680501353819 2023-04-03 05:55:54,536 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 60d32876aa6a41e4822ebddf04be2324, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680501353088 2023-04-03 05:55:54,536 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 25be44160438474782ad1131dc4f64fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680501353819 2023-04-03 05:55:54,548 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#B#compaction#485 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:54,548 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#A#compaction#484 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:54,562 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:54,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:54,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:54,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:54,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:54,565 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:54,565 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:54,565 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:54,565 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:54,566 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,566 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,567 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:54,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:54,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:54,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,568 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/5558a632322a4e6393ece52519a26b15 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/5558a632322a4e6393ece52519a26b15 2023-04-03 05:55:54,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,568 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:54,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:54,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:54,569 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:54,569 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:54,569 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:54,569 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:54,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501414554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501414555, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,571 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:54,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,572 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/A of 5d777b61781e184e82ec86ca4072f25a into 5558a632322a4e6393ece52519a26b15(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:54,572 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:54,572 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/A, priority=13, startTime=1680501354532; duration=0sec 2023-04-03 05:55:54,572 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,572 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,573 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37070 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,573 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/C is initiating minor compaction (all files) 2023-04-03 05:55:54,573 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/C in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:54,573 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/b3e9f04abf274013a5e27c9f5ee7eae4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/01fcbb95a1c5493fb14750a6ecf7cb27, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/41a5c3d05e58452b9ce53cc5ccb964df] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=36.2 K 2023-04-03 05:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67222bdb[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:54,573 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:54,573 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,575 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting b3e9f04abf274013a5e27c9f5ee7eae4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680501352731 2023-04-03 05:55:54,575 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 01fcbb95a1c5493fb14750a6ecf7cb27, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1680501353088 2023-04-03 05:55:54,575 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 41a5c3d05e58452b9ce53cc5ccb964df, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680501353819 2023-04-03 05:55:54,575 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:54,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:54,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:54,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,577 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:54,577 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,577 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:54,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:54,580 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:54,580 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:54,580 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:54,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:54,581 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:54,581 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#C#compaction#486 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:54,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501414570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,582 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,582 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,582 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:54,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:54,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501414570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,583 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:54,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:54,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:54,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:54,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:54,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:54,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:54,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:54,587 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:54,587 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:54,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:54,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:54,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,587 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501414671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501414671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,687 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:54,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:54,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501414683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501414684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:54,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:54,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:54,690 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:54,690 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:54,690 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:54,690 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:54,691 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,691 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,693 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:54,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:54,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:54,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:54,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:54,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:54,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:54,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:54,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:54,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:54,694 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:54,695 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:54,695 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:54,695 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:54,695 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:54,696 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:54,698 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1dee3be4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:54,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:54,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,699 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,699 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:54,699 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,700 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:54,699 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:54,700 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:54,701 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:54,701 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:54,701 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,701 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:54,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:54,703 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:54,703 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:54,703 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:54,703 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:54,703 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:54,705 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,705 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,705 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:54,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,706 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:54,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:54,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:54,709 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:54,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:54,710 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:54,710 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:54,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:54,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:54,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:54,810 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:54,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:54,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:54,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:54,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:54,813 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:54,813 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:54,813 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:54,813 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:54,813 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,813 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,814 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,814 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:54,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:54,815 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,815 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:54,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:54,816 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:54,816 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:54,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:54,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:54,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:54,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:54,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:54,816 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:54,816 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:54,816 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:54,816 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:54,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:54,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:54,816 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,817 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:54,817 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:54,818 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,818 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,818 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,818 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:54,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:54,820 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bfbb35f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:54,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:54,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,821 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,821 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:54,821 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:54,822 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:54,822 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,822 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:54,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:54,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:54,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:54,826 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:54,826 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:54,826 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:54,826 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:54,826 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:54,828 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,828 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,829 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,829 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:54,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,829 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:54,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:54,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:54,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:54,831 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:54,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,832 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:54,832 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,832 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,832 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,832 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:54,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:54,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:54,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:54,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501414874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,880 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501414875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501414889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:54,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501414890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,932 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:54,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:54,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:54,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:54,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:54,935 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:54,935 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:54,935 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:54,935 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:54,936 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,936 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,937 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:54,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:54,937 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,937 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:54,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:54,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:54,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:54,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:54,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:54,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:54,939 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:54,939 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:54,939 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:54,939 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:54,939 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:54,947 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,947 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,947 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,947 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:54,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:54,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5150bfd8[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:54,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:54,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,950 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/088e4a76e7f840cc8b4cf7961c1245b9 2023-04-03 05:55:54,953 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:54,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:54,953 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,954 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:54,955 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:54,957 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:54,957 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:54,957 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:54,957 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:54,957 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:54,958 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:54,961 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/24bba2960e3c4dacaa3121d36f118a3c 2023-04-03 05:55:54,961 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/247a0c05793142d7bf89a4f34afb0a38 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/247a0c05793142d7bf89a4f34afb0a38 2023-04-03 05:55:54,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:54,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,966 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/B of 5d777b61781e184e82ec86ca4072f25a into 247a0c05793142d7bf89a4f34afb0a38(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:54,966 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:54,966 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/B, priority=13, startTime=1680501354532; duration=0sec 2023-04-03 05:55:54,966 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:54,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:54,966 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:54,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:54,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,969 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:54,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:54,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,971 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,971 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,971 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:54,971 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:54,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:54,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:54,971 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:54,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:54,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:54,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:54,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:54,971 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:54,972 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:54,971 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:54,972 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,972 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:54,972 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:54,972 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:54,975 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/a66a03e17b424d6ebba1904e1bf5569c 2023-04-03 05:55:54,979 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/088e4a76e7f840cc8b4cf7961c1245b9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/088e4a76e7f840cc8b4cf7961c1245b9 2023-04-03 05:55:54,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/088e4a76e7f840cc8b4cf7961c1245b9, entries=250, sequenceid=375, filesize=16.7 K 2023-04-03 05:55:54,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/24bba2960e3c4dacaa3121d36f118a3c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/24bba2960e3c4dacaa3121d36f118a3c 2023-04-03 05:55:54,985 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/24bba2960e3c4dacaa3121d36f118a3c, entries=150, sequenceid=375, filesize=11.9 K 2023-04-03 05:55:54,988 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/a66a03e17b424d6ebba1904e1bf5569c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/a66a03e17b424d6ebba1904e1bf5569c 2023-04-03 05:55:54,992 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/3049535ef1844b8e9dec3f7fcaa11113 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/3049535ef1844b8e9dec3f7fcaa11113 2023-04-03 05:55:54,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/a66a03e17b424d6ebba1904e1bf5569c, entries=150, sequenceid=375, filesize=11.9 K 2023-04-03 05:55:54,992 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~275.95 KB/282576, currentSize=87.22 KB/89310 for 5d777b61781e184e82ec86ca4072f25a in 460ms, sequenceid=375, compaction requested=false 2023-04-03 05:55:54,992 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:54,995 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/C of 5d777b61781e184e82ec86ca4072f25a into 3049535ef1844b8e9dec3f7fcaa11113(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:54,995 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:54,995 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/C, priority=13, startTime=1680501354532; duration=0sec 2023-04-03 05:55:54,995 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:55,072 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:55,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:55,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:55,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:55,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:55,075 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:55,075 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:55,075 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:55,075 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:55,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,076 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,077 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:55,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:55,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:55,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:55,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:55,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:55,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:55,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:55,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:55,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:55,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:55,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:55,079 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 05:55:55,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:55,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:55,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:55,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:55,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:55,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:55,086 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/4fad471e68134f0dadecdc1e7f65e994 2023-04-03 05:55:55,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:55,186 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:55,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501415204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501415205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501415212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501415212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501415313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501415313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501415314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,335 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501415319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,501 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/727398b17b30420f81ce8e5a4d4cd3e7 2023-04-03 05:55:55,511 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=394 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/da81b1b14f5543c0b47a8503a8ccbcf5 2023-04-03 05:55:55,514 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/4fad471e68134f0dadecdc1e7f65e994 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/4fad471e68134f0dadecdc1e7f65e994 2023-04-03 05:55:55,517 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/4fad471e68134f0dadecdc1e7f65e994, entries=150, sequenceid=394, filesize=11.9 K 2023-04-03 05:55:55,518 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/727398b17b30420f81ce8e5a4d4cd3e7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/727398b17b30420f81ce8e5a4d4cd3e7 2023-04-03 05:55:55,521 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/727398b17b30420f81ce8e5a4d4cd3e7, entries=150, sequenceid=394, filesize=11.9 K 2023-04-03 05:55:55,521 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/da81b1b14f5543c0b47a8503a8ccbcf5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/da81b1b14f5543c0b47a8503a8ccbcf5 2023-04-03 05:55:55,525 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/da81b1b14f5543c0b47a8503a8ccbcf5, entries=150, sequenceid=394, filesize=11.9 K 2023-04-03 05:55:55,526 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 5d777b61781e184e82ec86ca4072f25a in 447ms, sequenceid=394, compaction requested=true 2023-04-03 05:55:55,526 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:55,526 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:55,526 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:55,528 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:55,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:55,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:55,528 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 05:55:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,528 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:55,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:55,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:55,528 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:55,528 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:55,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:55,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:55,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:55,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:55,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27da2ada[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:55,530 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:55,531 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,533 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,533 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:55,533 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:55,533 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:55,534 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,534 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:55,536 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:55,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:55,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:55,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:55,541 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:55,541 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:55,541 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:55,541 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:55,541 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:55,543 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,543 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:55,543 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:55,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:55,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:55,543 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:55,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,545 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:55,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:55,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:55,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:55,548 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:55,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:55,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:55,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:55,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:55,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,549 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:55,549 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:55,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,557 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501415548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501415552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501415552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501415558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,649 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:55,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:55,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:55,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:55,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:55,654 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:55,654 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:55,654 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:55,654 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:55,655 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,655 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,657 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:55,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:55,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,659 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:55,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,659 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:55,659 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,660 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,660 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:55,661 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:55,661 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:55,661 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:55,661 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:55,661 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:55,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501415659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501415661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,663 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:55,663 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:55,663 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501415663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,664 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:55,664 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:55,664 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:55,664 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:55,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:55,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501415665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,665 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,666 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:55,666 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,666 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:55,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:55,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:55,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b8704de[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:55,668 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:55,668 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,669 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,669 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:55,669 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:55,669 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,669 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:55,669 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:55,671 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:55,671 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:55,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:55,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:55,673 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:55,673 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:55,673 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:55,673 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:55,673 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:55,675 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,675 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:55,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:55,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,676 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:55,677 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:55,679 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:55,679 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:55,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:55,679 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,680 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:55,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:55,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:55,679 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:55,680 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:55,680 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:55,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:55,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:55,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:55,780 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:55,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:55,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:55,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:55,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:55,783 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:55,783 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:55,783 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:55,783 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:55,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,785 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:55,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:55,785 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,785 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,785 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:55,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:55,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:55,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:55,786 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:55,786 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:55,786 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:55,786 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:55,787 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:55,787 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:55,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,788 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:55,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:55,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:55,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d8027ba[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:55,790 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:55,790 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,792 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:55,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:55,792 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:55,794 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:55,794 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:55,794 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:55,794 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,794 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:55,794 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:55,794 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:55,794 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:55,794 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:55,794 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:55,794 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:55,796 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:55,796 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,796 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:55,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:55,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:55,797 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,798 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:55,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:55,800 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:55,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:55,801 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:55,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:55,801 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,801 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,801 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:55,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:55,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:55,801 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:55,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501415865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501415865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501415865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:55,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501415866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,901 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:55,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:55,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:55,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:55,916 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:55,916 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:55,916 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:55,916 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:55,917 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,917 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,919 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,919 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:55,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:55,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:55,919 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,919 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,920 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:55,924 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:55,924 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:55,928 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:55,928 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:55,928 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:55,928 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:55,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:55,930 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,930 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:55,930 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,930 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:55,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@797af3ad[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:55,931 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:55,931 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,932 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:55,933 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,933 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:55,933 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:55,934 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:55,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:55,935 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:55,935 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:55,935 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:55,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:55,938 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:55,938 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:55,938 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:55,938 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:55,938 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:55,939 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/9ecb0dd2e0e146d6b4c68db725f1d9d9 2023-04-03 05:55:55,940 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:55,940 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,941 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:55,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:55,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,942 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,944 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,946 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,946 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,946 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:55,946 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:55,946 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:55,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:55,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:55,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:55,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:55,947 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:55,947 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,947 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:55,948 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:55,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:55,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:55,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:55,948 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:55,948 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:56,048 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:56,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:56,050 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:56,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:56,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:56,051 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:56,051 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,051 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:56,051 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:56,052 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,052 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,053 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,053 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,053 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,053 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:56,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:56,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,054 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:56,054 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:56,054 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:56,055 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,055 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:56,055 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:56,055 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:56,055 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:56,057 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,057 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:56,058 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f008aa2[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:56,058 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:56,059 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,060 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,061 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:56,061 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:56,061 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,061 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:56,061 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:56,062 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,062 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:56,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:56,065 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:56,065 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:56,065 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:56,065 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:56,065 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:56,066 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,066 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,066 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:56,067 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,068 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:56,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:56,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:56,071 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:56,071 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:56,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,071 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,072 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:56,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:56,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:56,172 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:56,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:56,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:56,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:56,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:56,175 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:56,175 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,175 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:56,175 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:56,175 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,175 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,178 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,178 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:56,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501416174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,178 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,178 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:56,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:56,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501416175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:56,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501416176, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,179 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:56,179 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:56,179 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:56,180 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,180 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:56,180 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:56,180 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:56,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:56,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501416177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,182 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:56,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cd2ce68[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:56,183 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:56,183 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,184 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:56,185 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,185 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:56,186 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,186 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:56,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:56,188 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:56,188 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:56,188 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:56,188 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:56,188 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:56,193 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,193 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,193 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,194 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,195 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,197 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,197 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,198 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:56,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:56,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:56,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:56,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:56,198 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:56,198 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:56,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:56,298 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:56,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:56,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:56,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:56,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:56,301 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:56,301 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,301 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:56,301 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:56,302 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,302 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,303 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,303 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,303 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:56,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,304 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:56,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:56,305 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:56,305 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:56,305 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:56,305 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,305 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:56,305 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:56,306 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:56,306 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:56,307 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,307 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,307 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,307 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:56,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50439bbf[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:56,309 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:56,309 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,311 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:56,311 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:56,311 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,311 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:56,312 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:56,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:56,315 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:56,315 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:56,315 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:56,315 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:56,315 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:56,316 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,316 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,316 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:56,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,316 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,318 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,320 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,320 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,320 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:56,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,320 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:56,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:56,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:56,320 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:56,320 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,321 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,321 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:56,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:56,321 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,321 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:56,321 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:56,321 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,354 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/526450d71cfa4b99b1a48a4f93a6a457 2023-04-03 05:55:56,421 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:56,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:56,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:56,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:56,424 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:56,424 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,424 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:56,424 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:56,424 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,425 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,427 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,427 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,427 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,427 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:56,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:56,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:56,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:56,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:56,428 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:56,428 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,428 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:56,428 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:56,428 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:56,428 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,428 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:56,428 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:56,429 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:56,429 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:56,430 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,430 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,430 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,430 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:56,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:56,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e0afd81[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:56,432 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:56,432 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,432 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:56,433 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:56,433 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,433 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:56,434 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:56,434 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:56,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:56,437 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:56,437 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:56,437 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:56,437 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:56,437 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:56,437 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:56,439 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,439 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,439 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:56,440 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,441 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:56,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,443 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:56,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:56,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:56,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:56,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:56,443 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,444 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,444 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,444 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:56,444 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:56,544 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:56,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:56,546 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:56,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:56,547 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:56,547 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:56,547 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,547 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:56,547 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:56,548 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,548 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,549 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,549 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,549 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:56,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:56,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:56,551 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:56,551 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:56,551 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:56,551 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,551 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:56,551 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:56,551 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:56,551 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,551 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:56,551 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:56,551 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,551 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:56,551 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:56,552 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,552 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:56,552 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:56,553 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,553 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,554 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:56,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75d43d78[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:56,555 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:56,555 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,557 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,557 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:56,557 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:56,557 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:56,557 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,557 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:56,559 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,559 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:56,559 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,559 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:56,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:56,561 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:56,561 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:56,561 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:56,561 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:56,561 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:56,563 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,563 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,563 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:56,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,563 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,565 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,570 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,570 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,571 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:56,570 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:56,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:56,571 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:56,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:56,571 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:56,570 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,571 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:56,571 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:56,571 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,571 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,571 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,571 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,571 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,671 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:56,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:56,673 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:56,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:56,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:56,674 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:56,674 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,674 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:56,675 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:56,675 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,675 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,676 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:56,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:56,677 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:56,678 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:56,678 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,678 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:56,678 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:56,678 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:56,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:56,679 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,679 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:56,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f1f0c24[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:56,681 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:56,681 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,682 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,682 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:56,682 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:56,682 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,682 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:56,683 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:56,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501416679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,684 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,684 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:56,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,684 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:56,684 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:56,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501416680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:56,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501416682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:56,687 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:56,687 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:56,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:56,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:56,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:56,687 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:56,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:56,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501416683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,688 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,688 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:56,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,688 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:56,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:56,689 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,690 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:56,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:56,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:56,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:56,694 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,694 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:56,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:56,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:56,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:56,694 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,694 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:56,694 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:56,763 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/29bd891365d74dd8b6bc1f5023d7160b 2023-04-03 05:55:56,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/9ecb0dd2e0e146d6b4c68db725f1d9d9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9ecb0dd2e0e146d6b4c68db725f1d9d9 2023-04-03 05:55:56,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9ecb0dd2e0e146d6b4c68db725f1d9d9, entries=200, sequenceid=411, filesize=14.3 K 2023-04-03 05:55:56,771 DEBUG [MemStoreFlusher.0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-04-03 05:55:56,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/526450d71cfa4b99b1a48a4f93a6a457 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/526450d71cfa4b99b1a48a4f93a6a457 2023-04-03 05:55:56,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/526450d71cfa4b99b1a48a4f93a6a457, entries=150, sequenceid=411, filesize=11.9 K 2023-04-03 05:55:56,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/29bd891365d74dd8b6bc1f5023d7160b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/29bd891365d74dd8b6bc1f5023d7160b 2023-04-03 05:55:56,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/29bd891365d74dd8b6bc1f5023d7160b, entries=150, sequenceid=411, filesize=11.9 K 2023-04-03 05:55:56,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 5d777b61781e184e82ec86ca4072f25a in 1256ms, sequenceid=411, compaction requested=true 2023-04-03 05:55:56,784 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:56,784 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:56,784 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:56,784 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:55:56,784 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:55:56,784 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:56,785 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:55:56,785 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/B is initiating minor compaction (all files) 2023-04-03 05:55:56,785 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/B in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,786 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/247a0c05793142d7bf89a4f34afb0a38, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/24bba2960e3c4dacaa3121d36f118a3c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/727398b17b30420f81ce8e5a4d4cd3e7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/526450d71cfa4b99b1a48a4f93a6a457] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=48.2 K 2023-04-03 05:55:56,786 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56712 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:55:56,786 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/A is initiating minor compaction (all files) 2023-04-03 05:55:56,786 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/A in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,786 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/5558a632322a4e6393ece52519a26b15, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/088e4a76e7f840cc8b4cf7961c1245b9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/4fad471e68134f0dadecdc1e7f65e994, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9ecb0dd2e0e146d6b4c68db725f1d9d9] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=55.4 K 2023-04-03 05:55:56,786 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 247a0c05793142d7bf89a4f34afb0a38, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680501353819 2023-04-03 05:55:56,786 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 5558a632322a4e6393ece52519a26b15, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680501353819 2023-04-03 05:55:56,786 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 24bba2960e3c4dacaa3121d36f118a3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680501354514 2023-04-03 05:55:56,787 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 088e4a76e7f840cc8b4cf7961c1245b9, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680501354506 2023-04-03 05:55:56,787 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 727398b17b30420f81ce8e5a4d4cd3e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1680501354552 2023-04-03 05:55:56,787 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 4fad471e68134f0dadecdc1e7f65e994, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1680501354552 2023-04-03 05:55:56,787 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 526450d71cfa4b99b1a48a4f93a6a457, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680501355204 2023-04-03 05:55:56,787 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 9ecb0dd2e0e146d6b4c68db725f1d9d9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680501355204 2023-04-03 05:55:56,794 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:56,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:56,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:56,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:56,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:56,807 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:56,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:56,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:56,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,808 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#B#compaction#495 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:56,809 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,809 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:56,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:56,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,810 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:56,810 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:56,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:56,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:56,810 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:56,812 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#A#compaction#496 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:56,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:56,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:56,812 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:56,812 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,812 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:56,813 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 05:55:56,813 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:56,813 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:56,813 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:56,813 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:56,813 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:56,813 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:56,835 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/08c933df7d424cb1b0d6e3d9861cd4d4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/08c933df7d424cb1b0d6e3d9861cd4d4 2023-04-03 05:55:56,839 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/A of 5d777b61781e184e82ec86ca4072f25a into 08c933df7d424cb1b0d6e3d9861cd4d4(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:56,839 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:56,839 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/A, priority=12, startTime=1680501356784; duration=0sec 2023-04-03 05:55:56,839 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:56,839 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:55:56,840 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:55:56,840 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/C is initiating minor compaction (all files) 2023-04-03 05:55:56,840 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/C in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:56,840 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/3049535ef1844b8e9dec3f7fcaa11113, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/a66a03e17b424d6ebba1904e1bf5569c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/da81b1b14f5543c0b47a8503a8ccbcf5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/29bd891365d74dd8b6bc1f5023d7160b] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=48.2 K 2023-04-03 05:55:56,841 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 3049535ef1844b8e9dec3f7fcaa11113, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=345, earliestPutTs=1680501353819 2023-04-03 05:55:56,842 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting a66a03e17b424d6ebba1904e1bf5569c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680501354514 2023-04-03 05:55:56,842 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting da81b1b14f5543c0b47a8503a8ccbcf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=394, earliestPutTs=1680501354552 2023-04-03 05:55:56,845 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 29bd891365d74dd8b6bc1f5023d7160b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680501355204 2023-04-03 05:55:56,845 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/bcd6b6608cda4eae9aa426567f95b825 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/bcd6b6608cda4eae9aa426567f95b825 2023-04-03 05:55:56,849 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/B of 5d777b61781e184e82ec86ca4072f25a into bcd6b6608cda4eae9aa426567f95b825(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:56,850 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:56,850 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/B, priority=12, startTime=1680501356784; duration=0sec 2023-04-03 05:55:56,850 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:56,852 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/66c7c33d057b4e1ea53b2257fa1957e6 2023-04-03 05:55:56,853 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#C#compaction#498 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:56,872 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/edab102d7470499686ac5924ffae8dff as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/edab102d7470499686ac5924ffae8dff 2023-04-03 05:55:56,877 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/C of 5d777b61781e184e82ec86ca4072f25a into edab102d7470499686ac5924ffae8dff(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:56,877 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:56,877 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/C, priority=12, startTime=1680501356784; duration=0sec 2023-04-03 05:55:56,877 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:56,880 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/b6511c99251841eda4c0fed23f50fe4f 2023-04-03 05:55:57,289 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=426 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/eec06d3e96b140ebb8c171a2c15a81c0 2023-04-03 05:55:57,293 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/66c7c33d057b4e1ea53b2257fa1957e6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/66c7c33d057b4e1ea53b2257fa1957e6 2023-04-03 05:55:57,300 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/66c7c33d057b4e1ea53b2257fa1957e6, entries=150, sequenceid=426, filesize=11.9 K 2023-04-03 05:55:57,301 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/b6511c99251841eda4c0fed23f50fe4f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b6511c99251841eda4c0fed23f50fe4f 2023-04-03 05:55:57,304 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b6511c99251841eda4c0fed23f50fe4f, entries=150, sequenceid=426, filesize=11.9 K 2023-04-03 05:55:57,305 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/eec06d3e96b140ebb8c171a2c15a81c0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/eec06d3e96b140ebb8c171a2c15a81c0 2023-04-03 05:55:57,309 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/eec06d3e96b140ebb8c171a2c15a81c0, entries=150, sequenceid=426, filesize=11.9 K 2023-04-03 05:55:57,309 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 5d777b61781e184e82ec86ca4072f25a in 497ms, sequenceid=426, compaction requested=false 2023-04-03 05:55:57,310 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:57,310 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:57,310 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:57,310 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:57,310 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,310 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:57,310 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,313 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,313 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:57,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@534de21b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:57,315 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:57,315 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,316 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,316 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:57,316 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:57,316 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:57,316 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,316 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:57,318 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,318 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:57,321 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:57,321 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:57,321 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:57,321 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:57,321 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:57,321 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:57,323 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,323 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:57,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,323 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:57,323 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,325 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,331 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:57,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:57,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:57,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:57,331 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:57,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:57,331 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:57,331 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:57,331 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:57,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:57,431 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:57,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:57,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:57,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:57,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:57,435 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:57,435 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:57,435 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:57,435 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:57,435 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,435 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,436 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:57,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:57,436 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,437 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,437 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:57,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:57,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:57,438 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:57,438 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:57,438 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:57,438 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:57,438 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:57,438 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:57,439 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,440 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:57,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:57,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a82fb06[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:57,443 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:57,443 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,444 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:57,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:57,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:57,444 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,444 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:57,446 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,446 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,446 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:57,446 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:57,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:57,449 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:57,449 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:57,449 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:57,449 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:57,449 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:57,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,450 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:57,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,451 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,453 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,458 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:57,458 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,458 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:57,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:57,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:57,458 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:57,458 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:57,458 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:57,458 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:57,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:57,458 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,459 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,459 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,459 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,558 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:57,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:57,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:57,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:57,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:57,561 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:57,561 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:57,561 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:57,561 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:57,562 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,562 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,564 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,564 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:57,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:57,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:57,565 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,565 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:57,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:57,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,569 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:57,569 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:57,569 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:57,569 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:57,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:57,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:57,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:57,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:57,571 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:57,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ee936d3[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:57,573 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:57,573 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,574 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:57,574 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,574 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:57,574 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:57,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:57,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:57,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:57,578 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:57,578 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:57,578 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:57,578 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:57,578 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:57,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:57,581 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:57,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,582 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,585 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:57,585 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:57,585 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:57,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:57,585 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:57,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:57,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:57,585 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:57,585 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:57,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:57,585 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,585 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,585 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,585 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,585 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,585 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:57,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:57,685 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:57,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:57,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:57,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:57,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:57,692 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:57,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:57,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:57,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:57,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,694 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:57,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:57,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:57,695 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:55:57,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:57,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:57,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:57,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:57,695 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:57,695 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:57,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:57,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:57,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:57,696 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:57,696 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:57,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:57,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:57,697 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,698 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,699 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,699 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:57,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49f06106[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:57,701 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:57,701 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/9adf08cc22dc4c51acfde8b090169017 2023-04-03 05:55:57,707 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:57,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:57,707 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:57,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:57,708 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,708 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:57,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:57,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:57,710 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:57,711 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:57,711 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:57,711 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:57,711 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:57,712 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,712 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,712 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,713 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,714 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:57,716 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:57,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:57,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:57,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:57,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:57,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:57,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,716 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:57,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,717 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:57,717 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:57,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:57,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/ccd83e653a3d40f88dd0f28b068acd52 2023-04-03 05:55:57,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680501357741, max timestamp=1680501357750Num uniques 859; 2023-04-03 05:55:57,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:55:57,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680501357741, max timestamp=1680501357750Num uniques 859; 2023-04-03 05:55:57,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680501357741, max timestamp=1680501357750Num uniques 859; 2023-04-03 05:55:57,752 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:55:57,753 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:55:57,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:57,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501417773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,779 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:57,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501417773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:57,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501417774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,781 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:57,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501417779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,817 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:57,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:57,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:57,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:57,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:57,820 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:57,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:57,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:57,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:57,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,820 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,823 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:57,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:57,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:57,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:57,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:57,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:57,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:57,827 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:57,827 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:57,827 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:57,827 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:57,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:57,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:57,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:57,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:57,829 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,830 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,830 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,830 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:57,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:57,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75910f19[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:57,830 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:57,830 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,831 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:57,832 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:57,833 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:57,834 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,834 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:57,834 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:57,834 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:57,834 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:57,834 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:57,834 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:57,834 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:57,837 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,837 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,837 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:57,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,839 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,840 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:57,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:57,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,842 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:57,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:57,842 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:57,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:57,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:57,842 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:57,842 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:57,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:57,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:57,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:57,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501417880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:57,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501417881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:57,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501417882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:57,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501417882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,943 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:57,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:57,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:57,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:57,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:57,946 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:57,946 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:57,946 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:57,946 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:57,946 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,946 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,947 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:57,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:57,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:57,947 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,948 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:57,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:57,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:57,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:57,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:57,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:57,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:57,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:57,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:57,949 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:57,949 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:57,949 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:57,949 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:57,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:57,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:57,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:57,949 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:57,959 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,959 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,959 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,959 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:57,961 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18a69066[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:57,961 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:57,961 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,962 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,962 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,962 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:57,962 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:57,962 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:57,962 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:57,963 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:57,963 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:57,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:57,967 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:57,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:57,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:57,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:57,967 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:57,968 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:57,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:57,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:57,969 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:57,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:57,970 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:57,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:57,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,974 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:57,974 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:57,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:57,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:57,974 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:57,974 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:57,974 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,974 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,974 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:57,974 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:57,974 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:57,974 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:57,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:57,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:57,974 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:57,974 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,975 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:57,975 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:58,075 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:58,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:58,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:58,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:58,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:58,078 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:58,078 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:58,078 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:58,078 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:58,078 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:58,078 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:58,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:58,079 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:58,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:58,080 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,080 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:58,080 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:58,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:58,080 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:58,080 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:58,081 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:58,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:58,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:58,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:58,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:58,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:58,081 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:58,081 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:58,081 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:58,081 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:58,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:58,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:58,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:58,081 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:58,083 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,083 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:58,083 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:58,083 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:58,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:58,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:58,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:58,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f9e620f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:58,085 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:58,085 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:58,086 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:58,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:58,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:58,086 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:58,087 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,087 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:58,089 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:58,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:58,089 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:58,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:58,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:58,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:58,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:58,094 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:58,094 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:58,094 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:58,094 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:58,094 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:58,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501418088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,095 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:58,095 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:58,095 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:58,095 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:58,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:58,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501418090, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,096 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:58,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:58,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:58,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:58,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501418091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,097 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501418091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:58,098 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,101 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,101 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:58,101 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:58,101 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,101 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:58,101 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:58,101 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:58,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:58,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:58,101 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:58,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:58,101 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:58,101 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:58,101 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:58,101 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:58,101 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:58,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:58,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=441 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/51585e4b030b4c68af85c5aaf4cd4267 2023-04-03 05:55:58,133 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/9adf08cc22dc4c51acfde8b090169017 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9adf08cc22dc4c51acfde8b090169017 2023-04-03 05:55:58,137 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9adf08cc22dc4c51acfde8b090169017, entries=150, sequenceid=441, filesize=11.9 K 2023-04-03 05:55:58,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/ccd83e653a3d40f88dd0f28b068acd52 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/ccd83e653a3d40f88dd0f28b068acd52 2023-04-03 05:55:58,141 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/ccd83e653a3d40f88dd0f28b068acd52, entries=100, sequenceid=441, filesize=9.5 K 2023-04-03 05:55:58,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/51585e4b030b4c68af85c5aaf4cd4267 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/51585e4b030b4c68af85c5aaf4cd4267 2023-04-03 05:55:58,148 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/51585e4b030b4c68af85c5aaf4cd4267, entries=100, sequenceid=441, filesize=9.5 K 2023-04-03 05:55:58,149 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 5d777b61781e184e82ec86ca4072f25a in 454ms, sequenceid=441, compaction requested=true 2023-04-03 05:55:58,149 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:58,149 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:58,149 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:58,149 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:58,149 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:58,150 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:58,150 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37338 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:58,150 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34898 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:58,150 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/A is initiating minor compaction (all files) 2023-04-03 05:55:58,150 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/B is initiating minor compaction (all files) 2023-04-03 05:55:58,150 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/A in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:58,151 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/B in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:58,151 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/08c933df7d424cb1b0d6e3d9861cd4d4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/66c7c33d057b4e1ea53b2257fa1957e6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9adf08cc22dc4c51acfde8b090169017] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=36.5 K 2023-04-03 05:55:58,151 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/bcd6b6608cda4eae9aa426567f95b825, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b6511c99251841eda4c0fed23f50fe4f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/ccd83e653a3d40f88dd0f28b068acd52] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=34.1 K 2023-04-03 05:55:58,151 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 08c933df7d424cb1b0d6e3d9861cd4d4, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680501355204 2023-04-03 05:55:58,151 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting bcd6b6608cda4eae9aa426567f95b825, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680501355204 2023-04-03 05:55:58,151 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 66c7c33d057b4e1ea53b2257fa1957e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1680501355529 2023-04-03 05:55:58,151 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b6511c99251841eda4c0fed23f50fe4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1680501355529 2023-04-03 05:55:58,152 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ccd83e653a3d40f88dd0f28b068acd52, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680501357694 2023-04-03 05:55:58,152 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 9adf08cc22dc4c51acfde8b090169017, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680501357690 2023-04-03 05:55:58,159 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#A#compaction#504 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:58,173 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#B#compaction#505 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:58,201 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:58,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:58,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:58,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:58,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:58,216 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:58,216 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:58,216 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:58,216 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:58,216 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:58,217 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:58,218 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:58,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:58,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:58,218 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,218 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:58,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:58,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:58,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:58,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:58,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:58,220 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:58,221 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:58,222 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:58,222 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:58,222 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=140.89 KB heapSize=390.84 KB 2023-04-03 05:55:58,223 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:58,223 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:58,223 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:58,223 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:58,223 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:58,223 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:55:58,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:58,401 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:58,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501418408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501418409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501418410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501418418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501418519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,525 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501418519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,529 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501418520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,532 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501418527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,611 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/352655293e174972a5dd3572e98c8129 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/352655293e174972a5dd3572e98c8129 2023-04-03 05:55:58,615 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/581900d285084c0bb7014558981eb2ba as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/581900d285084c0bb7014558981eb2ba 2023-04-03 05:55:58,617 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/B of 5d777b61781e184e82ec86ca4072f25a into 352655293e174972a5dd3572e98c8129(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:58,617 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:58,617 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/B, priority=13, startTime=1680501358149; duration=0sec 2023-04-03 05:55:58,617 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:55:58,617 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:55:58,621 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:55:58,621 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/C is initiating minor compaction (all files) 2023-04-03 05:55:58,621 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/C in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:58,621 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/edab102d7470499686ac5924ffae8dff, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/eec06d3e96b140ebb8c171a2c15a81c0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/51585e4b030b4c68af85c5aaf4cd4267] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=34.0 K 2023-04-03 05:55:58,621 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting edab102d7470499686ac5924ffae8dff, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680501355204 2023-04-03 05:55:58,622 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting eec06d3e96b140ebb8c171a2c15a81c0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=426, earliestPutTs=1680501355529 2023-04-03 05:55:58,622 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 51585e4b030b4c68af85c5aaf4cd4267, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680501357694 2023-04-03 05:55:58,623 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/A of 5d777b61781e184e82ec86ca4072f25a into 581900d285084c0bb7014558981eb2ba(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:58,623 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:58,623 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/A, priority=13, startTime=1680501358149; duration=0sec 2023-04-03 05:55:58,623 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:58,627 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#C#compaction#507 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:55:58,646 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/d5194b0cbca14cb49af40de7c318e849 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/d5194b0cbca14cb49af40de7c318e849 2023-04-03 05:55:58,650 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/C of 5d777b61781e184e82ec86ca4072f25a into d5194b0cbca14cb49af40de7c318e849(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:55:58,650 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:58,650 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/C, priority=13, startTime=1680501358150; duration=0sec 2023-04-03 05:55:58,650 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:55:58,652 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/8bbbe2ae88934210b220318b1f93d71a 2023-04-03 05:55:58,735 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501418730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501418731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501418732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:58,737 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:58,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501418733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501419037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,038 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501419037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501419039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,044 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501419039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,079 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/fa0566e151064830b9413635e8f0d2de 2023-04-03 05:55:59,127 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=476 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/d52afe76a6c24d4f8aceb96fa4c31da0 2023-04-03 05:55:59,133 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/8bbbe2ae88934210b220318b1f93d71a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8bbbe2ae88934210b220318b1f93d71a 2023-04-03 05:55:59,138 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8bbbe2ae88934210b220318b1f93d71a, entries=150, sequenceid=476, filesize=11.9 K 2023-04-03 05:55:59,139 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/fa0566e151064830b9413635e8f0d2de as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/fa0566e151064830b9413635e8f0d2de 2023-04-03 05:55:59,142 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/fa0566e151064830b9413635e8f0d2de, entries=150, sequenceid=476, filesize=11.9 K 2023-04-03 05:55:59,143 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/d52afe76a6c24d4f8aceb96fa4c31da0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/d52afe76a6c24d4f8aceb96fa4c31da0 2023-04-03 05:55:59,147 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/d52afe76a6c24d4f8aceb96fa4c31da0, entries=150, sequenceid=476, filesize=11.9 K 2023-04-03 05:55:59,147 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~390.80 KB/400176, currentSize=46.96 KB/48090 for 5d777b61781e184e82ec86ca4072f25a in 925ms, sequenceid=476, compaction requested=false 2023-04-03 05:55:59,147 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:59,147 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:59,150 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,151 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35acc4ab[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:59,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:59,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,153 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,153 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:59,153 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:59,153 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:59,153 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:59,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:59,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:59,155 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:59,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:59,157 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:59,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:59,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:59,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:59,157 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:59,158 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,158 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,158 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:59,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:59,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,160 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,163 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,163 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,163 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:59,163 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:59,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:59,163 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:59,163 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:59,164 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:59,163 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:59,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:59,164 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:59,164 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:59,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:59,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,264 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:59,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:59,266 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:59,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:59,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:59,267 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:59,267 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:59,267 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:59,267 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:59,268 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,268 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,269 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,269 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:59,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:59,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:59,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:59,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:59,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:59,276 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:59,277 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:59,277 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:59,277 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:59,277 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:59,277 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:55:59,277 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:59,277 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:59,277 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:59,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:59,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:59,278 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:59,298 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/a60cd41d5d5f421fa88ccabee64c8606 2023-04-03 05:55:59,312 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/58dfb604dd8d4b039474134ca56ba864 2023-04-03 05:55:59,335 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=489 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/721e671b8afd4adea7abfa7a2bb7c2f3 2023-04-03 05:55:59,340 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/a60cd41d5d5f421fa88ccabee64c8606 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/a60cd41d5d5f421fa88ccabee64c8606 2023-04-03 05:55:59,344 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/a60cd41d5d5f421fa88ccabee64c8606, entries=150, sequenceid=489, filesize=11.9 K 2023-04-03 05:55:59,344 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/58dfb604dd8d4b039474134ca56ba864 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/58dfb604dd8d4b039474134ca56ba864 2023-04-03 05:55:59,349 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/58dfb604dd8d4b039474134ca56ba864, entries=150, sequenceid=489, filesize=11.9 K 2023-04-03 05:55:59,350 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/721e671b8afd4adea7abfa7a2bb7c2f3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/721e671b8afd4adea7abfa7a2bb7c2f3 2023-04-03 05:55:59,355 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/721e671b8afd4adea7abfa7a2bb7c2f3, entries=150, sequenceid=489, filesize=11.9 K 2023-04-03 05:55:59,356 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 5d777b61781e184e82ec86ca4072f25a in 79ms, sequenceid=489, compaction requested=true 2023-04-03 05:55:59,356 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:59,356 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:59,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:59,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:59,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:59,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:59,358 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,358 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:59,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48b8fcc0[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:59,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:59,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,365 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,365 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:59,365 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:59,365 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:59,365 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,365 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:59,366 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,366 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:59,366 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,366 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:59,369 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:59,369 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:59,369 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:59,369 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:59,369 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:59,369 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:59,371 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,371 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,371 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:59,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,371 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,373 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:59,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:59,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:59,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,375 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:59,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:59,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:59,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:59,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:59,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,376 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:59,376 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:59,376 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:59,476 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:59,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:59,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:59,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:59,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:59,479 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:59,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:59,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:59,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:59,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,482 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:59,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:59,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:59,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:59,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:59,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:59,483 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:59,483 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:59,483 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:59,491 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:59,491 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:59,491 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:59,491 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:59,491 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:55:59,491 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:59,491 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:59,492 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:59,492 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,492 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:59,492 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:59,493 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,493 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,494 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,494 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:59,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64b17011[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:59,495 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:59,495 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,496 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,497 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:59,497 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:59,497 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:59,498 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:59,498 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:59,498 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:59,503 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,503 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,503 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:59,506 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:59,506 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:59,506 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:59,506 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:59,506 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:59,506 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:59,509 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,509 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,509 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:59,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:59,510 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:59,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:59,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,514 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:59,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:59,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:59,515 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:59,515 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:59,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:59,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:59,555 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:55:59,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:55:59,556 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:55:59,557 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:59,557 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:55:59,557 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:59,557 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:55:59,557 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:55:59,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501359587, max timestamp=1680501359605Num uniques 859; 2023-04-03 05:55:59,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501359587, max timestamp=1680501359605Num uniques 859; 2023-04-03 05:55:59,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:55:59,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501359587, max timestamp=1680501359605Num uniques 859; 2023-04-03 05:55:59,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:55:59,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:55:59,615 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:59,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:59,617 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:59,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:59,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:59,619 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:59,619 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:59,619 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:59,619 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:59,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,621 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:59,621 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:59,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:59,622 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,622 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:59,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:59,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,622 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:59,636 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:59,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:59,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:59,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:59,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:59,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:59,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:59,638 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:59,638 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:59,638 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:59,638 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:59,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:59,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:59,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:59,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:59,639 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:59,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68b3e892[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:59,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:59,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,647 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,647 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:59,647 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:59,647 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:59,647 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,647 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:59,648 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,648 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:59,648 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,648 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:59,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:59,650 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:59,650 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:59,650 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:59,650 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:59,651 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:59,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:59,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,655 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,659 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,659 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:59,659 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,660 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:59,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:59,660 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:59,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:59,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:59,660 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,660 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,660 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,660 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:59,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:59,660 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:59,660 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:59,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:59,660 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,663 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501419646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501419653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501419663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,676 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501419663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,760 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:59,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:59,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:59,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:59,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:59,763 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:59,763 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:59,763 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:59,763 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:59,763 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,763 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:59,765 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:59,766 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,766 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:59,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:59,767 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:59,767 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:59,767 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:59,767 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:59,767 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:59,767 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:59,767 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:59,769 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,769 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,769 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,769 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:59,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3be9e0b3[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:59,770 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:59,770 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,771 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:59,771 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,771 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:59,771 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:59,771 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:59,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501419764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501419765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,776 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:59,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:59,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:59,778 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:59,778 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:59,778 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:59,778 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:59,778 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:59,780 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,780 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,780 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:59,781 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,782 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,783 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501419777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501419777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,784 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:59,784 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:59,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:59,784 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,784 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:59,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:59,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:59,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:59,784 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,784 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,785 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,785 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,785 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,784 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:59,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:59,785 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:59,785 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:59,885 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:55:59,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:55:59,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:55:59,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:55:59,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:55:59,888 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:55:59,888 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:59,888 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:55:59,888 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:55:59,888 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,888 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,889 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:59,889 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:59,890 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,890 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:55:59,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:55:59,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:55:59,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:55:59,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:55:59,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:55:59,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:55:59,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:55:59,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:55:59,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:55:59,891 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:59,891 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:55:59,891 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:55:59,891 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:55:59,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:55:59,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:55:59,892 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,892 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:55:59,892 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:55:59,894 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,894 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,895 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,895 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:55:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:55:59,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@343845f2[Count = 0] remaining members to acquire global barrier 2023-04-03 05:55:59,896 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:55:59,896 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,897 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:55:59,897 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:55:59,897 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:55:59,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:55:59,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,899 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:55:59,899 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,899 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:55:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:55:59,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:55:59,902 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:55:59,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:55:59,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:55:59,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:55:59,902 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:55:59,903 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,903 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:55:59,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:55:59,903 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:59,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:59,904 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:55:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:55:59,905 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:55:59,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:55:59,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:55:59,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:55:59,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:55:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:55:59,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:55:59,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:55:59,908 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:55:59,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:55:59,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:55:59,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:55:59,908 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:55:59,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:55:59,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:55:59,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:55:59,908 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:55:59,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501419974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501419975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501419985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:55:59,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:55:59,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501419986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,008 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:00,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:00,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:00,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:00,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:00,012 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:00,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:00,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:00,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,013 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,014 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,014 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:00,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:00,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:00,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:00,015 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,015 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:00,015 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:00,015 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:00,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:00,017 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:00,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:00,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5be69a9c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:00,019 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:00,019 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:00,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:00,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:00,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:00,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/cee876bb351941d88fd1164b3a6db6e8 2023-04-03 05:56:00,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:00,023 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:00,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:00,026 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:00,026 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:00,026 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:00,026 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:00,026 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:00,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,029 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,032 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,035 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,035 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,035 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,035 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:00,035 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:00,035 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,035 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,035 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,035 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,035 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,035 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,035 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:00,035 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:00,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,135 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:00,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:00,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:00,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:00,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:00,138 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:00,138 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,139 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:00,139 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:00,139 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,139 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,140 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,140 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,141 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:00,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,142 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:00,142 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:00,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:00,142 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,142 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:00,142 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:00,142 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:00,142 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:00,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:00,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c661e0a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:00,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:00,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,148 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:00,148 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:00,148 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,148 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:00,149 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,149 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:00,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:00,157 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:00,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:00,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:00,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:00,157 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:00,158 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,158 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,158 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,160 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,165 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:00,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:00,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:00,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,165 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,166 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,166 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:00,166 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,166 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:00,166 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,266 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:00,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:00,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:00,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:00,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:00,274 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:00,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:00,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:00,275 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,275 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,276 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,276 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,276 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,276 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:00,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:00,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:00,277 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:00,277 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,277 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:00,277 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:00,277 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:00,278 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:00,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,279 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,279 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:00,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58fe8f4d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:00,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:00,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,282 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,282 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:00,282 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:00,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:00,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,283 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:00,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:00,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:00,288 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:00,288 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:00,288 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:00,288 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:00,288 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:00,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:00,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501420281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,290 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,290 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,292 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:00,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:00,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,294 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:00,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,294 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:00,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,294 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,294 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:00,295 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:00,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501420288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:00,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501420291, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:00,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501420292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,395 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:00,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:00,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:00,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:00,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:00,398 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:00,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:00,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:00,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,399 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,399 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,401 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:00,401 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:00,401 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:00,402 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,402 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:00,402 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:00,402 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:00,402 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:00,404 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,405 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,405 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:00,406 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e9f4675[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:00,406 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:00,406 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,407 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:00,408 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,408 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:00,408 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:00,409 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,409 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,409 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,409 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:00,412 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:00,412 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:00,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:00,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:00,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:00,412 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:00,413 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,413 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,413 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,414 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,415 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,418 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,418 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,418 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:00,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:00,418 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,418 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,419 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:00,419 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:00,418 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,418 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,419 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,419 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,419 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/6b3bd689de4b4a0ba228d8d94e8ef178 2023-04-03 05:56:00,519 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:00,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:00,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:00,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:00,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:00,522 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:00,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:00,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:00,523 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,523 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,524 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,524 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,524 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:00,525 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:00,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:00,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:00,527 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:00,527 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,527 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:00,527 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:00,527 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:00,527 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:00,529 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:00,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41c20b40[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:00,530 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:00,530 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,532 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,532 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:00,533 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:00,534 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:00,536 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:00,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:00,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:00,539 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:00,539 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:00,539 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:00,539 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:00,539 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:00,539 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:00,540 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,540 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,541 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,542 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,543 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,548 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:00,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:00,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,549 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:00,549 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:00,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,649 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:00,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:00,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:00,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:00,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:00,652 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:00,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:00,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:00,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,655 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,655 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,655 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,655 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:00,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:00,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:00,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:00,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:00,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:00,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:00,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:00,656 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,656 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:00,656 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:00,656 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:00,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:00,656 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,657 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:00,657 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,658 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,658 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:00,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:00,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d48db3c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:00,668 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:00,668 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,669 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:00,670 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:00,678 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,678 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:00,678 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:00,678 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,679 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:00,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:00,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:00,682 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:00,683 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:00,683 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:00,683 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:00,683 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:00,685 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,685 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,685 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,685 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,687 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,689 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,689 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,689 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,689 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,689 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,689 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,689 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,689 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,689 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,689 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:00,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:00,690 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:00,690 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:00,790 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:00,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:00,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:00,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:00,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:00,794 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:00,795 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,795 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:00,795 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:00,795 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,795 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,796 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,796 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,796 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,797 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:00,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,797 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:00,798 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:00,798 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:00,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:00,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:00,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:00,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:00,798 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,798 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:00,798 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:00,798 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:00,798 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:00,799 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,799 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:00,799 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:00,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,800 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:00,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:00,802 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7160b33f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:00,802 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:00,802 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:00,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501420794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:00,804 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501420800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,804 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,805 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:00,804 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:00,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501420801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:00,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:00,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:00,807 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,807 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:00,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,807 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,807 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:00,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:00,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:00,809 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:00,809 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:00,809 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:00,809 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:00,809 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:00,811 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,811 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:00,811 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,811 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:00,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501420802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:00,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:00,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,815 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:00,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:00,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:00,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:00,815 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:00,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,815 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:00,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,815 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:00,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:00,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:00,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:00,815 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,841 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=501 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/dcba649505c945ceafedb7edf3223d7c 2023-04-03 05:56:00,844 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/cee876bb351941d88fd1164b3a6db6e8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/cee876bb351941d88fd1164b3a6db6e8 2023-04-03 05:56:00,847 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/cee876bb351941d88fd1164b3a6db6e8, entries=200, sequenceid=501, filesize=14.3 K 2023-04-03 05:56:00,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/6b3bd689de4b4a0ba228d8d94e8ef178 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/6b3bd689de4b4a0ba228d8d94e8ef178 2023-04-03 05:56:00,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/6b3bd689de4b4a0ba228d8d94e8ef178, entries=150, sequenceid=501, filesize=11.9 K 2023-04-03 05:56:00,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/dcba649505c945ceafedb7edf3223d7c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/dcba649505c945ceafedb7edf3223d7c 2023-04-03 05:56:00,854 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/dcba649505c945ceafedb7edf3223d7c, entries=150, sequenceid=501, filesize=11.9 K 2023-04-03 05:56:00,855 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 5d777b61781e184e82ec86ca4072f25a in 1300ms, sequenceid=501, compaction requested=true 2023-04-03 05:56:00,855 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:00,855 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:00,855 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:00,855 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:00,855 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:00,855 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:00,856 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52070 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:00,856 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/A is initiating minor compaction (all files) 2023-04-03 05:56:00,856 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49630 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:00,856 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/A in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,856 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/B is initiating minor compaction (all files) 2023-04-03 05:56:00,856 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/581900d285084c0bb7014558981eb2ba, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8bbbe2ae88934210b220318b1f93d71a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/a60cd41d5d5f421fa88ccabee64c8606, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/cee876bb351941d88fd1164b3a6db6e8] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=50.8 K 2023-04-03 05:56:00,856 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/B in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,856 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/352655293e174972a5dd3572e98c8129, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/fa0566e151064830b9413635e8f0d2de, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/58dfb604dd8d4b039474134ca56ba864, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/6b3bd689de4b4a0ba228d8d94e8ef178] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=48.5 K 2023-04-03 05:56:00,857 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 581900d285084c0bb7014558981eb2ba, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680501355529 2023-04-03 05:56:00,857 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 352655293e174972a5dd3572e98c8129, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680501355529 2023-04-03 05:56:00,857 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting fa0566e151064830b9413635e8f0d2de, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1680501357766 2023-04-03 05:56:00,857 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8bbbe2ae88934210b220318b1f93d71a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1680501357766 2023-04-03 05:56:00,857 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 58dfb604dd8d4b039474134ca56ba864, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680501358405 2023-04-03 05:56:00,858 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting a60cd41d5d5f421fa88ccabee64c8606, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680501358405 2023-04-03 05:56:00,858 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6b3bd689de4b4a0ba228d8d94e8ef178, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1680501359550 2023-04-03 05:56:00,858 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting cee876bb351941d88fd1164b3a6db6e8, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1680501359546 2023-04-03 05:56:00,863 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#B#compaction#516 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:00,864 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#A#compaction#517 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:00,875 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/5a485a7c7b1047a58854c1e61dc2b983 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/5a485a7c7b1047a58854c1e61dc2b983 2023-04-03 05:56:00,878 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/A of 5d777b61781e184e82ec86ca4072f25a into 5a485a7c7b1047a58854c1e61dc2b983(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:00,878 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:00,878 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/A, priority=12, startTime=1680501360855; duration=0sec 2023-04-03 05:56:00,878 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:00,878 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:00,879 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49596 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:00,879 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/C is initiating minor compaction (all files) 2023-04-03 05:56:00,879 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/C in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,879 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/d5194b0cbca14cb49af40de7c318e849, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/d52afe76a6c24d4f8aceb96fa4c31da0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/721e671b8afd4adea7abfa7a2bb7c2f3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/dcba649505c945ceafedb7edf3223d7c] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=48.4 K 2023-04-03 05:56:00,880 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting d5194b0cbca14cb49af40de7c318e849, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=441, earliestPutTs=1680501355529 2023-04-03 05:56:00,880 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting d52afe76a6c24d4f8aceb96fa4c31da0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=476, earliestPutTs=1680501357766 2023-04-03 05:56:00,880 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 721e671b8afd4adea7abfa7a2bb7c2f3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=489, earliestPutTs=1680501358405 2023-04-03 05:56:00,881 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting dcba649505c945ceafedb7edf3223d7c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1680501359550 2023-04-03 05:56:00,887 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#C#compaction#518 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:00,893 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/7e26d10ca08b42f0acd5b70e64965c9f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/7e26d10ca08b42f0acd5b70e64965c9f 2023-04-03 05:56:00,896 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/C of 5d777b61781e184e82ec86ca4072f25a into 7e26d10ca08b42f0acd5b70e64965c9f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:00,896 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:00,896 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/C, priority=12, startTime=1680501360855; duration=0sec 2023-04-03 05:56:00,897 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:00,915 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:00,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:00,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:00,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:00,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:00,918 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:00,918 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,918 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:00,918 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:00,919 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,919 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,920 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:00,920 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:00,921 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:00,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,921 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:00,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:00,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:00,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:00,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:00,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:00,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:00,922 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:00,922 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:00,922 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 05:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:56:00,923 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:01,273 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/f1b28944c64e48d68363c10863df4cca as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f1b28944c64e48d68363c10863df4cca 2023-04-03 05:56:01,277 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/B of 5d777b61781e184e82ec86ca4072f25a into f1b28944c64e48d68363c10863df4cca(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:01,277 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:01,278 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/B, priority=12, startTime=1680501360855; duration=0sec 2023-04-03 05:56:01,278 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:01,337 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/ff03fb83a558409ab69765720ec22396 2023-04-03 05:56:01,746 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/1041926654ce4e8dacc56e93220fd85a 2023-04-03 05:56:01,760 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=540 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/069e0067168242aca4ca2d31a08584d8 2023-04-03 05:56:01,763 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/ff03fb83a558409ab69765720ec22396 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ff03fb83a558409ab69765720ec22396 2023-04-03 05:56:01,766 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ff03fb83a558409ab69765720ec22396, entries=150, sequenceid=540, filesize=11.9 K 2023-04-03 05:56:01,767 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/1041926654ce4e8dacc56e93220fd85a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1041926654ce4e8dacc56e93220fd85a 2023-04-03 05:56:01,770 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1041926654ce4e8dacc56e93220fd85a, entries=150, sequenceid=540, filesize=11.9 K 2023-04-03 05:56:01,771 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/069e0067168242aca4ca2d31a08584d8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/069e0067168242aca4ca2d31a08584d8 2023-04-03 05:56:01,774 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/069e0067168242aca4ca2d31a08584d8, entries=150, sequenceid=540, filesize=11.9 K 2023-04-03 05:56:01,774 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 5d777b61781e184e82ec86ca4072f25a in 852ms, sequenceid=540, compaction requested=false 2023-04-03 05:56:01,774 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:01,774 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:01,774 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:01,776 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:01,777 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,777 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:01,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,779 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:01,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:01,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:01,780 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2518ef88[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:01,780 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:01,780 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,781 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:01,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:01,781 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,781 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:01,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:01,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:01,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:01,783 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:01,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:01,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:01,785 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:01,786 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:01,786 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:01,786 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:01,786 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:01,788 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,788 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:01,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:01,789 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,789 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:01,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:01,790 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:01,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:01,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:01,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:01,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:01,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:01,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:01,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:01,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:01,800 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:01,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:01,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:01,800 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:01,800 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:01,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:56:01,821 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:01,821 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:56:01,821 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:01,821 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:56:01,821 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:01,821 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:56:01,821 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:01,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501361829, max timestamp=1680501361858Num uniques 859; 2023-04-03 05:56:01,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:01,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501361829, max timestamp=1680501361858Num uniques 859; 2023-04-03 05:56:01,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:01,860 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501361829, max timestamp=1680501361858Num uniques 859; 2023-04-03 05:56:01,861 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:01,901 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:01,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:01,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:01,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:01,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501421898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:01,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:01,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:01,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501421901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,913 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:01,913 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:01,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:01,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501421902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,913 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:01,913 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:01,914 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,914 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:01,918 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:01,918 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:01,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501421906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:01,918 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,918 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:01,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:01,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:01,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:01,919 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:01,919 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:01,919 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:01,919 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:01,919 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:01,919 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:01,920 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:01,920 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:01,920 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:01,920 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:01,920 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:01,927 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,927 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:01,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:01,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:01,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:01,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@74d17860[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:01,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:01,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,931 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:01,931 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:01,931 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,931 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:01,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:01,933 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:01,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:01,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:01,935 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:01,935 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:01,935 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:01,935 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:01,935 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:01,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:01,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:01,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:01,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,938 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:01,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:01,940 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:01,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,945 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,945 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:01,945 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:01,945 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:01,945 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:01,945 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:01,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:01,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:01,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:01,945 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:01,946 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:01,946 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,946 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:01,946 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:01,946 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:01,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:01,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:02,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501422014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:02,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501422014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:02,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501422014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:02,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501422020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,046 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:02,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:02,048 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:02,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:02,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:02,050 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:02,050 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,050 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:02,050 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:02,051 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,052 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,053 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,055 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,055 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,055 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:02,056 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:02,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:02,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:02,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:02,057 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:02,058 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,058 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:02,058 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:02,058 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:02,058 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:02,059 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,059 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,060 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:02,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f5bdcfd[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:02,064 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:02,064 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,065 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,065 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,066 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:02,066 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:02,066 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:02,066 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:02,067 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,067 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,067 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,067 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:02,071 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:02,071 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:02,071 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:02,071 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:02,071 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:02,071 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:02,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,072 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,073 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,074 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,078 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,078 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,078 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:02,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:02,078 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,078 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:02,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,078 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:02,078 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,078 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,079 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,079 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:02,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,179 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:02,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:02,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:02,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:02,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:02,182 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:02,182 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,182 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:02,182 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:02,183 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,183 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,184 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,184 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,185 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,185 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:02,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:02,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:02,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:02,186 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:02,186 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,186 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:02,187 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:02,187 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:02,187 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,187 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:02,187 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:02,187 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:02,188 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:02,189 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,189 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,189 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,189 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:02,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:02,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b566362[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:02,190 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:02,191 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,192 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:02,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:02,192 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:02,192 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,192 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:02,193 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:02,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:02,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:02,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:02,196 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:02,196 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:02,196 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:02,196 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:02,196 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:02,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,199 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,200 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,202 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,202 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,202 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,202 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,202 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,203 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,203 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,203 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,203 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,203 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:02,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:02,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,203 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:02,203 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:02,203 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:02,223 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:02,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501422220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:02,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501422221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,224 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:02,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501422221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,228 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/719e9000596249168d919b778c0741f9 2023-04-03 05:56:02,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:02,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501422228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,303 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:02,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:02,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:02,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:02,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:02,307 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:02,307 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,307 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:02,307 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:02,307 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,308 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,310 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:02,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,311 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:02,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,311 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:02,311 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:02,311 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:02,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:02,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:02,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:02,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:02,312 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,312 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:02,312 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:02,312 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:02,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:02,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:02,312 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:02,314 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,314 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,314 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,314 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:02,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:02,316 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d0bb2f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:02,316 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:02,316 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,325 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,326 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:02,326 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:02,326 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:02,326 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,326 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:02,328 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:02,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,328 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:02,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:02,331 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:02,331 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:02,331 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:02,331 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:02,331 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:02,332 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,332 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,332 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,334 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,341 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,341 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,341 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,341 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:02,341 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:02,341 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,341 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,342 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:02,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,342 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:02,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,442 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:02,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:02,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:02,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:02,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:02,445 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:02,445 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,445 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:02,445 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:02,445 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,445 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,446 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,446 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,447 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,447 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:02,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:02,448 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:02,448 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:02,448 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:02,448 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,448 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:02,449 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:02,449 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:02,449 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,449 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:02,449 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:02,450 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:02,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:02,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:02,450 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:02,451 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,451 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,452 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:02,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63dde0d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:02,453 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:02,453 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,454 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,454 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:02,454 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:02,454 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:02,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:02,456 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:02,456 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,456 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,456 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:02,458 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:02,458 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:02,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:02,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:02,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:02,459 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:02,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,460 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,460 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,461 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,463 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:02,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:02,463 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,463 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,464 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,464 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,464 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,464 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:02,464 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,464 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,464 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:02,464 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,464 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,465 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:02,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:02,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501422525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,528 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:02,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501422525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:02,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501422527, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,541 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:02,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501422536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,564 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:02,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:02,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:02,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:02,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:02,567 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:02,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:02,567 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:02,568 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,568 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,569 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,569 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,569 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,569 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:02,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:02,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:02,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:02,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:02,570 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:02,573 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,574 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:02,575 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37304603[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:02,575 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:02,575 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:02,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:02,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:02,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:02,577 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:02,577 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:02,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:02,580 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:02,580 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:02,580 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:02,580 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:02,580 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:02,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,583 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,584 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,589 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,589 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,589 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,589 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,589 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,589 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,589 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:02,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:02,589 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,589 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,589 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,590 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:02,590 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:02,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/fe94a23986d8473d9ec509204502ce50 2023-04-03 05:56:02,652 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/efcf7837650249b382c3be075bc3a92c 2023-04-03 05:56:02,656 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/719e9000596249168d919b778c0741f9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/719e9000596249168d919b778c0741f9 2023-04-03 05:56:02,659 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/719e9000596249168d919b778c0741f9, entries=200, sequenceid=551, filesize=14.3 K 2023-04-03 05:56:02,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/fe94a23986d8473d9ec509204502ce50 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/fe94a23986d8473d9ec509204502ce50 2023-04-03 05:56:02,664 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/fe94a23986d8473d9ec509204502ce50, entries=150, sequenceid=551, filesize=11.9 K 2023-04-03 05:56:02,665 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/efcf7837650249b382c3be075bc3a92c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/efcf7837650249b382c3be075bc3a92c 2023-04-03 05:56:02,668 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/efcf7837650249b382c3be075bc3a92c, entries=150, sequenceid=551, filesize=11.9 K 2023-04-03 05:56:02,669 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 5d777b61781e184e82ec86ca4072f25a in 848ms, sequenceid=551, compaction requested=true 2023-04-03 05:56:02,669 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:02,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:02,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:02,669 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:02,669 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:02,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:02,670 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40020 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:02,670 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/A is initiating minor compaction (all files) 2023-04-03 05:56:02,670 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/A in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,670 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/5a485a7c7b1047a58854c1e61dc2b983, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ff03fb83a558409ab69765720ec22396, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/719e9000596249168d919b778c0741f9] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=39.1 K 2023-04-03 05:56:02,671 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37580 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:02,671 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/B is initiating minor compaction (all files) 2023-04-03 05:56:02,671 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 5a485a7c7b1047a58854c1e61dc2b983, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1680501359550 2023-04-03 05:56:02,671 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/B in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,671 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f1b28944c64e48d68363c10863df4cca, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1041926654ce4e8dacc56e93220fd85a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/fe94a23986d8473d9ec509204502ce50] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=36.7 K 2023-04-03 05:56:02,671 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting ff03fb83a558409ab69765720ec22396, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1680501359645 2023-04-03 05:56:02,672 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 719e9000596249168d919b778c0741f9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680501361812 2023-04-03 05:56:02,672 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting f1b28944c64e48d68363c10863df4cca, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1680501359550 2023-04-03 05:56:02,672 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1041926654ce4e8dacc56e93220fd85a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1680501359645 2023-04-03 05:56:02,672 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting fe94a23986d8473d9ec509204502ce50, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680501361816 2023-04-03 05:56:02,683 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#B#compaction#525 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:02,683 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#A#compaction#526 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:02,690 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:02,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:02,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:02,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:02,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:02,693 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:02,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:02,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:02,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,694 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,695 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,695 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,695 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,695 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:02,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,696 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:02,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:02,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:02,697 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 05:56:02,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:56:02,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:02,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:56:02,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:02,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:56:02,697 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:02,708 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/e11ad43d0a8f47789bbc48012329ae6b 2023-04-03 05:56:02,717 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/7ed32784b7df4610ab054230a043ec15 2023-04-03 05:56:02,726 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=588 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/7ad068c39e8d452da18f4d5dc3c42d93 2023-04-03 05:56:02,729 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/e11ad43d0a8f47789bbc48012329ae6b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/e11ad43d0a8f47789bbc48012329ae6b 2023-04-03 05:56:02,732 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/e11ad43d0a8f47789bbc48012329ae6b, entries=150, sequenceid=588, filesize=11.9 K 2023-04-03 05:56:02,732 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/7ed32784b7df4610ab054230a043ec15 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/7ed32784b7df4610ab054230a043ec15 2023-04-03 05:56:02,736 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/7ed32784b7df4610ab054230a043ec15, entries=150, sequenceid=588, filesize=11.9 K 2023-04-03 05:56:02,738 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/7ad068c39e8d452da18f4d5dc3c42d93 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/7ad068c39e8d452da18f4d5dc3c42d93 2023-04-03 05:56:02,741 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/7ad068c39e8d452da18f4d5dc3c42d93, entries=150, sequenceid=588, filesize=11.9 K 2023-04-03 05:56:02,742 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for 5d777b61781e184e82ec86ca4072f25a in 45ms, sequenceid=588, compaction requested=true 2023-04-03 05:56:02,742 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:02,742 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:02,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:02,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:02,742 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:02,743 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,743 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,744 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,744 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:02,745 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f557ef4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:02,745 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:02,745 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,747 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,747 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:02,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,747 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:02,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:02,747 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:02,753 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,753 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:02,753 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,753 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:02,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:02,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:02,755 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:02,755 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:02,755 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:02,755 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:02,755 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:02,756 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,756 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,756 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,758 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,761 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:02,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:02,761 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:02,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,761 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:02,761 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:02,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,862 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:02,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:02,863 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:02,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:02,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:02,864 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:02,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,865 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:02,865 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:02,865 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,865 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,866 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,866 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,867 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,867 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:02,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:02,868 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:02,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:02,868 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,868 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:02,868 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:02,868 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:02,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:02,871 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,871 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,871 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,871 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:02,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63a0b7f2[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:02,873 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:02,873 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,874 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,874 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:02,874 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:02,874 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:02,874 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,874 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:02,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,875 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,876 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,876 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:02,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:02,880 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:02,880 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:02,880 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:02,880 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:02,880 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:02,880 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:02,881 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,881 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:02,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,881 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:02,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,882 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:02,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,896 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:02,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:02,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:02,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:02,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,899 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:02,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:02,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:02,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:02,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:02,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:02,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:02,899 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:02,899 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:02,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:02,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:02,999 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:03,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:03,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:03,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:03,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:03,008 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:03,008 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,008 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:03,008 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:03,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,009 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,011 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,011 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,011 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,011 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:03,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,011 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:03,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,012 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:03,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:03,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:03,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:03,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:03,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:03,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:03,016 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,016 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:03,016 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:03,016 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:03,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:03,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:03,016 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:03,018 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,019 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,019 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:03,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b05d03d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:03,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:03,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,023 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:03,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:03,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:03,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:03,025 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:03,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,025 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:03,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:03,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:03,028 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:03,028 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:03,028 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:03,028 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:03,028 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:03,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,030 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,030 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:03,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,030 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,033 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:03,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:03,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:03,037 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:03,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:03,037 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:03,037 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:03,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,037 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,038 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:03,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:03,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:56:03,051 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:03,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:56:03,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:03,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:56:03,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:03,052 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:56:03,052 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:03,060 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/f5c79682e49e4926b559910722adcb66 2023-04-03 05:56:03,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/66af248aa1f248dd807bbfedf6e3dce3 2023-04-03 05:56:03,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680501363070, max timestamp=1680501363095Num uniques 859; 2023-04-03 05:56:03,097 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:56:03,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680501363070, max timestamp=1680501363095Num uniques 859; 2023-04-03 05:56:03,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-03 05:56:03,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680501363070, max timestamp=1680501363095Num uniques 859; 2023-04-03 05:56:03,098 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-03 05:56:03,099 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/cb9cf11934ba43c494bc1a4c680ae801 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/cb9cf11934ba43c494bc1a4c680ae801 2023-04-03 05:56:03,099 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/73170c6d919f425c898fb718ec681a0b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/73170c6d919f425c898fb718ec681a0b 2023-04-03 05:56:03,102 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/B of 5d777b61781e184e82ec86ca4072f25a into 73170c6d919f425c898fb718ec681a0b(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:03,103 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:03,103 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/B, priority=13, startTime=1680501362669; duration=0sec 2023-04-03 05:56:03,103 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:03,103 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:03,104 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/A of 5d777b61781e184e82ec86ca4072f25a into cb9cf11934ba43c494bc1a4c680ae801(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:03,104 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:03,104 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49732 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:03,104 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/A, priority=13, startTime=1680501362669; duration=0sec 2023-04-03 05:56:03,104 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/C is initiating minor compaction (all files) 2023-04-03 05:56:03,104 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:03,104 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/C in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,104 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/7e26d10ca08b42f0acd5b70e64965c9f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/069e0067168242aca4ca2d31a08584d8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/efcf7837650249b382c3be075bc3a92c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/7ad068c39e8d452da18f4d5dc3c42d93] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=48.6 K 2023-04-03 05:56:03,105 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 7e26d10ca08b42f0acd5b70e64965c9f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=501, earliestPutTs=1680501359550 2023-04-03 05:56:03,105 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 069e0067168242aca4ca2d31a08584d8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=540, earliestPutTs=1680501359645 2023-04-03 05:56:03,105 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting efcf7837650249b382c3be075bc3a92c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680501361816 2023-04-03 05:56:03,105 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 7ad068c39e8d452da18f4d5dc3c42d93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680501361891 2023-04-03 05:56:03,122 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#C#compaction#533 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:03,137 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:03,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:03,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:03,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:03,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:03,146 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:03,146 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,146 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:03,146 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:03,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501423144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501423147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501423147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,156 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501423148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:03,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,157 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:03,158 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,158 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:03,160 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:03,160 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:03,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:03,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:03,168 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,168 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:03,168 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:03,168 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:03,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:03,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:03,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:03,170 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,170 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,170 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,170 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:03,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:03,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eb09f00[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:03,172 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:03,172 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,173 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,173 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:03,173 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:03,173 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,174 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:03,173 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:03,175 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,175 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:03,175 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:03,175 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:03,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:03,177 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:03,177 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:03,178 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:03,178 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:03,178 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:03,178 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:03,180 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,180 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:03,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,180 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:03,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:03,185 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,186 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,191 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,191 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:03,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:03,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:03,192 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,192 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,192 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:03,192 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:03,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:03,193 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:03,193 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:03,192 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,194 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,194 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,194 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,264 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501423255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501423257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501423258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501423258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,293 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:03,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:03,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:03,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:03,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:03,316 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:03,316 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,316 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:03,316 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:03,317 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,317 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,318 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,318 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,319 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,319 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:03,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:03,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,319 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:03,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:03,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:03,320 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:03,320 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,320 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:03,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:03,323 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:03,342 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,342 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:03,342 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:03,342 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:03,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:03,345 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,345 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,346 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,346 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:03,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@aa6e4fc[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:03,347 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:03,347 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,348 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:03,348 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:03,349 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:03,349 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,349 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:03,350 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:03,350 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,350 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,350 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:03,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:03,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:03,354 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:03,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:03,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:03,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:03,354 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:03,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:03,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:03,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,358 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:03,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:03,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:03,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:03,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,361 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:03,361 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:03,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:03,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:03,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:03,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,363 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,461 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:03,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:03,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:03,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:03,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:03,465 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:03,465 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,465 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:03,465 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:03,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,467 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,467 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,467 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,467 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:03,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:03,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:03,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:03,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:03,469 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,469 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:03,469 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:03,469 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:03,469 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:03,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,471 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501423467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501423467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501423468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501423468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,473 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:03,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:03,473 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c5fcd7d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:03,473 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:03,473 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,474 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:03,474 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,474 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:03,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:03,475 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=601 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/c241f5fda9dc4cebabd8f9df40c2ec8d 2023-04-03 05:56:03,477 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,477 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:03,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:03,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,481 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/f5c79682e49e4926b559910722adcb66 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/f5c79682e49e4926b559910722adcb66 2023-04-03 05:56:03,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:03,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:03,482 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:03,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:03,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:03,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:03,482 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:03,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,484 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,485 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:03,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/f5c79682e49e4926b559910722adcb66, entries=150, sequenceid=601, filesize=11.9 K 2023-04-03 05:56:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,486 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,487 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/66af248aa1f248dd807bbfedf6e3dce3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/66af248aa1f248dd807bbfedf6e3dce3 2023-04-03 05:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:03,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,489 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,490 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:03,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:03,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:03,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:03,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:03,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:03,492 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:03,492 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:03,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/66af248aa1f248dd807bbfedf6e3dce3, entries=150, sequenceid=601, filesize=11.9 K 2023-04-03 05:56:03,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/c241f5fda9dc4cebabd8f9df40c2ec8d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/c241f5fda9dc4cebabd8f9df40c2ec8d 2023-04-03 05:56:03,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/c241f5fda9dc4cebabd8f9df40c2ec8d, entries=150, sequenceid=601, filesize=11.9 K 2023-04-03 05:56:03,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=131.94 KB/135110 for 5d777b61781e184e82ec86ca4072f25a in 446ms, sequenceid=601, compaction requested=true 2023-04-03 05:56:03,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:03,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:03,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:03,497 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:03,497 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 05:56:03,498 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:03,498 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/A is initiating minor compaction (all files) 2023-04-03 05:56:03,498 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/A in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,498 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/cb9cf11934ba43c494bc1a4c680ae801, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/e11ad43d0a8f47789bbc48012329ae6b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/f5c79682e49e4926b559910722adcb66] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=36.8 K 2023-04-03 05:56:03,498 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting cb9cf11934ba43c494bc1a4c680ae801, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680501361816 2023-04-03 05:56:03,499 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e11ad43d0a8f47789bbc48012329ae6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680501361891 2023-04-03 05:56:03,499 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting f5c79682e49e4926b559910722adcb66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680501363041 2023-04-03 05:56:03,503 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#A#compaction#534 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:03,514 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/7cf5124f5a5d4dcea966f8854f7b597b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/7cf5124f5a5d4dcea966f8854f7b597b 2023-04-03 05:56:03,526 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/A of 5d777b61781e184e82ec86ca4072f25a into 7cf5124f5a5d4dcea966f8854f7b597b(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:03,526 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:03,526 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/A, priority=13, startTime=1680501363497; duration=0sec 2023-04-03 05:56:03,526 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 05:56:03,526 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 4 compacting, 1 eligible, 16 blocking 2023-04-03 05:56:03,527 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-03 05:56:03,527 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-03 05:56:03,527 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. because compaction request was cancelled 2023-04-03 05:56:03,527 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:03,528 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37682 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:03,528 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/B is initiating minor compaction (all files) 2023-04-03 05:56:03,528 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/B in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,528 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/73170c6d919f425c898fb718ec681a0b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/7ed32784b7df4610ab054230a043ec15, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/66af248aa1f248dd807bbfedf6e3dce3] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=36.8 K 2023-04-03 05:56:03,529 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 73170c6d919f425c898fb718ec681a0b, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1680501361816 2023-04-03 05:56:03,529 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 7ed32784b7df4610ab054230a043ec15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680501361891 2023-04-03 05:56:03,530 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 66af248aa1f248dd807bbfedf6e3dce3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680501363041 2023-04-03 05:56:03,537 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#B#compaction#535 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:03,548 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/f5371569c10a4914942a610c280566fc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/f5371569c10a4914942a610c280566fc 2023-04-03 05:56:03,550 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/423bb3fdb0bd4de9b2863c4f8e4b1989 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/423bb3fdb0bd4de9b2863c4f8e4b1989 2023-04-03 05:56:03,552 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/C of 5d777b61781e184e82ec86ca4072f25a into f5371569c10a4914942a610c280566fc(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:03,552 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:03,552 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/C, priority=12, startTime=1680501362669; duration=0sec 2023-04-03 05:56:03,552 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:03,555 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/B of 5d777b61781e184e82ec86ca4072f25a into 423bb3fdb0bd4de9b2863c4f8e4b1989(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:03,555 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:03,555 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/B, priority=13, startTime=1680501363497; duration=0sec 2023-04-03 05:56:03,555 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:03,592 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:03,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:03,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:03,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:03,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:03,595 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:03,595 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,595 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:03,595 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:03,595 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,597 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,597 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,597 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,597 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:03,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:03,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:03,598 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:03,598 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:03,598 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:03,598 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,600 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:03,600 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=131.94 KB heapSize=365.84 KB 2023-04-03 05:56:03,600 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:56:03,600 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:03,600 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:56:03,600 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:03,600 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:56:03,600 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:03,606 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/290b908d6f1344a5af4bf89eef3bf570 2023-04-03 05:56:03,645 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/f8cdbda990654f15bdf9bdab3e7766db 2023-04-03 05:56:03,706 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=640 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/c3782cf296784415aeb2be460940b8d0 2023-04-03 05:56:03,710 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/290b908d6f1344a5af4bf89eef3bf570 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/290b908d6f1344a5af4bf89eef3bf570 2023-04-03 05:56:03,713 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/290b908d6f1344a5af4bf89eef3bf570, entries=150, sequenceid=640, filesize=11.9 K 2023-04-03 05:56:03,714 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/f8cdbda990654f15bdf9bdab3e7766db as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f8cdbda990654f15bdf9bdab3e7766db 2023-04-03 05:56:03,718 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f8cdbda990654f15bdf9bdab3e7766db, entries=150, sequenceid=640, filesize=11.9 K 2023-04-03 05:56:03,719 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/c3782cf296784415aeb2be460940b8d0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/c3782cf296784415aeb2be460940b8d0 2023-04-03 05:56:03,722 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/c3782cf296784415aeb2be460940b8d0, entries=150, sequenceid=640, filesize=11.9 K 2023-04-03 05:56:03,722 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~131.94 KB/135110, heapSize ~365.80 KB/374576, currentSize=0 B/0 for 5d777b61781e184e82ec86ca4072f25a in 122ms, sequenceid=640, compaction requested=true 2023-04-03 05:56:03,723 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:03,723 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,723 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:03,723 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:03,723 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,723 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:03,723 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:03,726 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,726 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,726 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,726 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:03,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:03,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@497e7cf3[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:03,728 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:03,728 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,729 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:03,729 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:03,730 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,730 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:03,732 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:03,736 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:03,736 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:03,736 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:03,736 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:03,736 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:03,736 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:03,737 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,737 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:03,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,738 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,742 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,745 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,750 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,750 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,750 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,750 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,750 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,750 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:03,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:03,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:03,750 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,751 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,751 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:03,750 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:03,751 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:03,751 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:03,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:56:03,781 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:03,782 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:56:03,782 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:03,782 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:56:03,782 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:03,782 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:56:03,782 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:03,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501423828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501423829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501363797, max timestamp=1680501363840Num uniques 859; 2023-04-03 05:56:03,842 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:03,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501363797, max timestamp=1680501363840Num uniques 859; 2023-04-03 05:56:03,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:03,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501363797, max timestamp=1680501363840Num uniques 859; 2023-04-03 05:56:03,843 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:03,851 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:03,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:03,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:03,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:03,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:03,854 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:03,854 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,854 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:03,854 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:03,855 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,855 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,856 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,856 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,856 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,856 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:03,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,858 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:03,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:03,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:03,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:03,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,859 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:03,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:03,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:03,864 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,864 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:03,864 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:03,864 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:03,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:03,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:03,864 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:03,866 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,867 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:03,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@738cb243[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:03,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:03,868 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,868 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:03,869 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,869 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:03,869 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,869 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:03,869 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:03,870 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:03,872 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:03,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:03,872 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:03,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:03,874 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:03,874 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:03,874 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:03,874 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:03,874 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:03,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,875 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:03,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:03,877 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,877 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,881 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,881 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:03,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:03,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:03,881 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,882 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:03,881 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,882 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,882 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,882 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:03,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,882 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,882 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,882 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,882 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:03,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:03,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:03,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:03,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501423899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501423900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501423943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:03,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501423943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,982 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:03,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:03,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:03,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:03,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:03,985 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:03,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:03,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:03,986 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,986 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,987 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:03,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:03,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:03,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:03,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:03,988 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:03,988 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:03,988 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:03,988 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:03,993 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,993 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:03,993 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:03,993 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:03,993 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:03,996 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,996 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:03,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:03,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:03,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:03,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:03,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:03,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:03,998 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4eb039ca[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:03,998 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:03,998 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,999 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:03,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,000 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:04,000 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:04,000 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:04,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:04,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,001 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:04,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:04,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501424004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:04,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501424008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:04,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:04,013 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:04,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:04,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:04,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:04,013 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:04,015 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,015 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,015 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:04,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:04,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:04,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:04,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:04,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:04,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:04,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:04,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:04,021 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:04,021 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:04,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:04,121 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:04,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:04,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:04,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:04,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:04,125 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:04,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:04,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:04,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:04,126 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,126 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,127 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:04,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:04,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,128 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:04,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:04,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:04,133 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:04,133 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:04,133 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:04,133 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:04,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:04,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:04,135 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:04,135 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:04,135 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:04,135 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:04,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:04,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:04,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:04,135 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:04,137 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,137 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,138 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:04,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:04,139 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c163144[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:04,139 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:04,139 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,140 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,140 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:04,141 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,141 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:04,141 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:04,141 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:04,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:04,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:04,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:04,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:04,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:04,148 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:04,148 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:04,148 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:04,148 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:04,148 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:04,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,153 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,156 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:04,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:04,156 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:04,157 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:04,157 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:04,156 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:04,157 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:04,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:04,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:04,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:04,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:04,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:04,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:04,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:04,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501424151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501424157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,194 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/711560989f0447d8b98944d7e86be49e 2023-04-03 05:56:04,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:04,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501424213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:04,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501424215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,230 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:04,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:34638 deadline: 1680501424230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,232 DEBUG [Thread-3286] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18267 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:56:04,257 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:04,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:04,259 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:04,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:04,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:04,260 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:04,260 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:04,260 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:04,260 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:04,261 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,261 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,265 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:04,265 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:04,266 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,266 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:04,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:04,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:04,267 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:04,267 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:04,267 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:04,267 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:04,267 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:04,267 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:04,267 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:04,267 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:04,267 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:04,268 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:04,268 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:04,268 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:04,269 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,269 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,269 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,269 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:04,271 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66b547d1[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:04,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:04,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,272 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,272 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:04,272 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:04,272 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:04,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:04,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:04,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:04,275 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,276 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:04,276 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:04,276 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:04,276 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:04,276 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:04,276 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:04,276 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:04,282 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,282 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:04,282 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,282 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,284 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:04,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:04,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:04,286 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:04,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,286 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:04,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:04,287 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:04,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:04,287 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:04,287 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:04,387 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:04,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:04,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:04,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:04,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:04,391 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:04,391 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:04,391 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:04,391 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:04,391 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,391 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,393 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:04,393 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:04,393 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,393 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:04,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:04,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:04,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:04,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:04,394 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:04,394 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:04,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:04,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:04,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:04,395 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:04,395 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:04,395 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:04,395 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:04,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:04,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:04,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:04,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:04,397 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:04,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:04,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70bdffee[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:04,399 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:04,399 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,401 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,401 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:04,401 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:04,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:04,401 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:04,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,403 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:04,403 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:04,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,403 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:04,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:04,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:04,408 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:04,408 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:04,408 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:04,408 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:04,408 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:04,409 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,409 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,410 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:04,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,413 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:04,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:04,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:04,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,415 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:04,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:04,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:04,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,415 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:04,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:04,415 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:04,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:04,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34614 deadline: 1680501424461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:04,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34662 deadline: 1680501424462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,516 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:04,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:04,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:04,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:04,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:04,526 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:04,526 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:04,526 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:04,526 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:04,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:04,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34586 deadline: 1680501424520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,528 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:04,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:34608 deadline: 1680501424522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,529 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:04,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:04,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:04,529 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,530 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,530 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:04,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:04,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:04,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:04,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:04,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:04,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:04,540 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:04,540 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:04,540 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:04,540 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:04,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:04,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:04,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:04,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:04,543 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,543 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,543 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,543 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:04,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:04,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f8ce65a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:04,546 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:04,546 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,548 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,548 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:04,548 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:04,548 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:04,548 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,548 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:04,549 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:04,549 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:04,549 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:04,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:04,556 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:04,556 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:04,556 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:04,556 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:04,557 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:04,557 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:04,559 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,559 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:04,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:04,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:04,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,561 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:04,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:04,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:04,564 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:04,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,565 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:04,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,567 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,568 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,568 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,568 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:04,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:04,568 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:04,568 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,568 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:04,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:04,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:04,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:04,568 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,568 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:04,568 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:04,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:04,569 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:04,569 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:04,609 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/eb716506efed4bbcb7200ff00375fa93 2023-04-03 05:56:04,636 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=651 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/4026d200b76e4d3cbc957d88e8c968b7 2023-04-03 05:56:04,642 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/711560989f0447d8b98944d7e86be49e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/711560989f0447d8b98944d7e86be49e 2023-04-03 05:56:04,645 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/711560989f0447d8b98944d7e86be49e, entries=250, sequenceid=651, filesize=16.7 K 2023-04-03 05:56:04,646 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/eb716506efed4bbcb7200ff00375fa93 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/eb716506efed4bbcb7200ff00375fa93 2023-04-03 05:56:04,655 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/eb716506efed4bbcb7200ff00375fa93, entries=150, sequenceid=651, filesize=11.9 K 2023-04-03 05:56:04,657 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/4026d200b76e4d3cbc957d88e8c968b7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/4026d200b76e4d3cbc957d88e8c968b7 2023-04-03 05:56:04,667 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/4026d200b76e4d3cbc957d88e8c968b7, entries=150, sequenceid=651, filesize=11.9 K 2023-04-03 05:56:04,668 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5d777b61781e184e82ec86ca4072f25a in 887ms, sequenceid=651, compaction requested=true 2023-04-03 05:56:04,668 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:04,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:04,668 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:04,668 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:04,668 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:04,669 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:04,669 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42664 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:04,670 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:04,670 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/A is initiating minor compaction (all files) 2023-04-03 05:56:04,670 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/A in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:04,670 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/7cf5124f5a5d4dcea966f8854f7b597b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/290b908d6f1344a5af4bf89eef3bf570, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/711560989f0447d8b98944d7e86be49e] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=41.7 K 2023-04-03 05:56:04,670 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:04,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:04,670 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 7cf5124f5a5d4dcea966f8854f7b597b, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680501363041 2023-04-03 05:56:04,670 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/B is initiating minor compaction (all files) 2023-04-03 05:56:04,671 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/B in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:04,671 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/423bb3fdb0bd4de9b2863c4f8e4b1989, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f8cdbda990654f15bdf9bdab3e7766db, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/eb716506efed4bbcb7200ff00375fa93] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=36.9 K 2023-04-03 05:56:04,671 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 290b908d6f1344a5af4bf89eef3bf570, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1680501363134 2023-04-03 05:56:04,671 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 423bb3fdb0bd4de9b2863c4f8e4b1989, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680501363041 2023-04-03 05:56:04,671 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 711560989f0447d8b98944d7e86be49e, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1680501363776 2023-04-03 05:56:04,672 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting f8cdbda990654f15bdf9bdab3e7766db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1680501363134 2023-04-03 05:56:04,672 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting eb716506efed4bbcb7200ff00375fa93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1680501363777 2023-04-03 05:56:04,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:04,693 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#A#compaction#542 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:04,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:04,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:04,696 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:04,696 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:04,696 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:04,696 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:04,696 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,696 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,697 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:04,697 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:04,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:04,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:04,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:04,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,700 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:04,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:04,700 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:04,706 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#B#compaction#543 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:04,708 DEBUG [Thread-3299] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15f74c78 to 127.0.0.1:63744 2023-04-03 05:56:04,708 DEBUG [Thread-3299] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:04,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:04,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:04,712 DEBUG [Thread-3303] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f6e865f to 127.0.0.1:63744 2023-04-03 05:56:04,712 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:04,713 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:04,713 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:04,713 DEBUG [Thread-3303] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:04,714 DEBUG [Thread-3301] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1401cac1 to 127.0.0.1:63744 2023-04-03 05:56:04,714 DEBUG [Thread-3301] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:04,715 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:04,715 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:04,715 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:04,716 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. started... 2023-04-03 05:56:04,716 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-03 05:56:04,716 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:56:04,716 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:04,716 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:56:04,716 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:04,716 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:56:04,716 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:04,717 DEBUG [Thread-3297] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1f65627a to 127.0.0.1:63744 2023-04-03 05:56:04,717 DEBUG [Thread-3297] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:04,720 DEBUG [Thread-3295] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7a4e3b2e to 127.0.0.1:63744 2023-04-03 05:56:04,720 DEBUG [Thread-3295] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:04,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:56:04,971 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. as already flushing 2023-04-03 05:56:04,971 DEBUG [Thread-3284] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6e0b6187 to 127.0.0.1:63744 2023-04-03 05:56:04,971 DEBUG [Thread-3284] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:04,974 DEBUG [Thread-3292] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50150525 to 127.0.0.1:63744 2023-04-03 05:56:04,974 DEBUG [Thread-3292] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:05,034 DEBUG [Thread-3290] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3816bdf5 to 127.0.0.1:63744 2023-04-03 05:56:05,034 DEBUG [Thread-3288] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x15840296 to 127.0.0.1:63744 2023-04-03 05:56:05,034 DEBUG [Thread-3288] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:05,034 DEBUG [Thread-3290] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:05,136 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/f04c5484dbf44b29bec07b9cf7070f1a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f04c5484dbf44b29bec07b9cf7070f1a 2023-04-03 05:56:05,136 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/d53e714e40f04498b6dc8ead1e1e314f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/d53e714e40f04498b6dc8ead1e1e314f 2023-04-03 05:56:05,140 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/9685d408dbbc42acbad623750a1541dd 2023-04-03 05:56:05,140 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/B of 5d777b61781e184e82ec86ca4072f25a into f04c5484dbf44b29bec07b9cf7070f1a(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:05,140 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:05,140 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/B, priority=13, startTime=1680501364668; duration=0sec 2023-04-03 05:56:05,140 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/A of 5d777b61781e184e82ec86ca4072f25a into d53e714e40f04498b6dc8ead1e1e314f(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:05,140 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:05,140 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:05,140 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/A, priority=13, startTime=1680501364668; duration=0sec 2023-04-03 05:56:05,140 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:05,140 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:05,142 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:05,142 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 5d777b61781e184e82ec86ca4072f25a/C is initiating minor compaction (all files) 2023-04-03 05:56:05,142 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5d777b61781e184e82ec86ca4072f25a/C in TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:05,142 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/f5371569c10a4914942a610c280566fc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/c241f5fda9dc4cebabd8f9df40c2ec8d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/c3782cf296784415aeb2be460940b8d0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/4026d200b76e4d3cbc957d88e8c968b7] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp, totalSize=48.7 K 2023-04-03 05:56:05,142 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting f5371569c10a4914942a610c280566fc, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=588, earliestPutTs=1680501361891 2023-04-03 05:56:05,143 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting c241f5fda9dc4cebabd8f9df40c2ec8d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=601, earliestPutTs=1680501363041 2023-04-03 05:56:05,143 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting c3782cf296784415aeb2be460940b8d0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=640, earliestPutTs=1680501363134 2023-04-03 05:56:05,143 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 4026d200b76e4d3cbc957d88e8c968b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=651, earliestPutTs=1680501363777 2023-04-03 05:56:05,150 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5d777b61781e184e82ec86ca4072f25a#C#compaction#546 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:05,157 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/dcee3a93424f4a6da11cfecf313a3a20 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/dcee3a93424f4a6da11cfecf313a3a20 2023-04-03 05:56:05,161 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5d777b61781e184e82ec86ca4072f25a/C of 5d777b61781e184e82ec86ca4072f25a into dcee3a93424f4a6da11cfecf313a3a20(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:05,161 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:05,161 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a., storeName=5d777b61781e184e82ec86ca4072f25a/C, priority=12, startTime=1680501364669; duration=0sec 2023-04-03 05:56:05,161 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:05,550 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/d8e6d0be91b54c8a8720a5f38e251bf6 2023-04-03 05:56:05,558 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=686 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/edaa1e2f03bd41ac924df0c34ea214e4 2023-04-03 05:56:05,561 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/9685d408dbbc42acbad623750a1541dd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9685d408dbbc42acbad623750a1541dd 2023-04-03 05:56:05,565 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9685d408dbbc42acbad623750a1541dd, entries=150, sequenceid=686, filesize=11.9 K 2023-04-03 05:56:05,565 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/d8e6d0be91b54c8a8720a5f38e251bf6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/d8e6d0be91b54c8a8720a5f38e251bf6 2023-04-03 05:56:05,573 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/d8e6d0be91b54c8a8720a5f38e251bf6, entries=150, sequenceid=686, filesize=11.9 K 2023-04-03 05:56:05,574 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/edaa1e2f03bd41ac924df0c34ea214e4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/edaa1e2f03bd41ac924df0c34ea214e4 2023-04-03 05:56:05,578 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/edaa1e2f03bd41ac924df0c34ea214e4, entries=150, sequenceid=686, filesize=11.9 K 2023-04-03 05:56:05,579 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=26.84 KB/27480 for 5d777b61781e184e82ec86ca4072f25a in 863ms, sequenceid=686, compaction requested=false 2023-04-03 05:56:05,579 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:05,579 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:05,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:05,586 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,586 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:05,587 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:05,587 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:05,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:05,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52fd6cfe[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:05,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:05,588 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:05,589 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:05,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:05,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:05,589 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:05,589 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:05,589 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:05,589 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,589 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:05,590 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,590 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:05,590 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:05,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:05,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:05,590 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:05,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:05,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:05,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:05,592 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:05,592 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:05,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:05,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:05,592 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:05,592 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:05,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:05,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:05,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:05,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:05,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:05,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:05,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:05,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:05,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:05,595 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:05,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:05,595 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:05,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:05,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:05,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:05,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:05,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,599 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:05,599 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:05,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:05,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:05,599 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,599 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:05,599 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:05,599 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:05,599 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:05,599 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:05,599 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:05,599 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:05,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:05,600 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:05,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:05,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:05,600 DEBUG [Thread-3294] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:05,600 DEBUG [Thread-3294] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:05,700 DEBUG [Thread-3294] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:05,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:14,285 DEBUG [Thread-3286] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2ab56e54 to 127.0.0.1:63744 2023-04-03 05:56:14,285 DEBUG [Thread-3286] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:14,285 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 05:56:14,285 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 118 2023-04-03 05:56:14,285 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 23 2023-04-03 05:56:14,285 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 170 2023-04-03 05:56:14,285 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 123 2023-04-03 05:56:14,285 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 132 2023-04-03 05:56:14,285 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 05:56:14,285 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 05:56:14,285 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 1926 2023-04-03 05:56:14,286 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 5778 rows 2023-04-03 05:56:14,286 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 1922 2023-04-03 05:56:14,286 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 5766 rows 2023-04-03 05:56:14,286 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 1920 2023-04-03 05:56:14,286 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 5759 rows 2023-04-03 05:56:14,286 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 1933 2023-04-03 05:56:14,286 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 5797 rows 2023-04-03 05:56:14,286 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(402): scanned 1929 2023-04-03 05:56:14,286 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(403): verified 5786 rows 2023-04-03 05:56:14,286 INFO [Listener at localhost/37275] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 05:56:14,286 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66369677 to 127.0.0.1:63744 2023-04-03 05:56:14,286 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:14,289 INFO [Listener at localhost/37275] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 05:56:14,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-03 05:56:14,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:14,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 05:56:14,294 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501374293"}]},"ts":"1680501374293"} 2023-04-03 05:56:14,294 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 05:56:14,296 INFO [PEWorker-4] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 05:56:14,296 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5d777b61781e184e82ec86ca4072f25a, UNASSIGN}] 2023-04-03 05:56:14,297 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5d777b61781e184e82ec86ca4072f25a, UNASSIGN 2023-04-03 05:56:14,298 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=5d777b61781e184e82ec86ca4072f25a, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:14,298 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501374298"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501374298"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501374298"}]},"ts":"1680501374298"} 2023-04-03 05:56:14,299 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:56:14,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 05:56:14,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:56:14,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5d777b61781e184e82ec86ca4072f25a, disabling compactions & flushes 2023-04-03 05:56:14,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:14,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:14,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. after waiting 0 ms 2023-04-03 05:56:14,450 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:14,450 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5d777b61781e184e82ec86ca4072f25a 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-03 05:56:14,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=A 2023-04-03 05:56:14,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:14,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=B 2023-04-03 05:56:14,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:14,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5d777b61781e184e82ec86ca4072f25a, store=C 2023-04-03 05:56:14,451 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:14,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 05:56:14,856 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=697 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/8f4f6c8e217d433c8e3f9557d8a357b5 2023-04-03 05:56:14,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=697 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/1d927709ded846ffbc3ea1ac7a1d76f6 2023-04-03 05:56:14,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 05:56:15,281 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=697 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/cfc40488d8404cb5b1a0490cff1f754f 2023-04-03 05:56:15,285 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/A/8f4f6c8e217d433c8e3f9557d8a357b5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8f4f6c8e217d433c8e3f9557d8a357b5 2023-04-03 05:56:15,289 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8f4f6c8e217d433c8e3f9557d8a357b5, entries=150, sequenceid=697, filesize=11.9 K 2023-04-03 05:56:15,289 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/B/1d927709ded846ffbc3ea1ac7a1d76f6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1d927709ded846ffbc3ea1ac7a1d76f6 2023-04-03 05:56:15,297 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1d927709ded846ffbc3ea1ac7a1d76f6, entries=150, sequenceid=697, filesize=11.9 K 2023-04-03 05:56:15,298 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/.tmp/C/cfc40488d8404cb5b1a0490cff1f754f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cfc40488d8404cb5b1a0490cff1f754f 2023-04-03 05:56:15,301 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cfc40488d8404cb5b1a0490cff1f754f, entries=150, sequenceid=697, filesize=11.9 K 2023-04-03 05:56:15,302 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 5d777b61781e184e82ec86ca4072f25a in 852ms, sequenceid=697, compaction requested=true 2023-04-03 05:56:15,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9cea537836f14288a89cbd2207ee44d0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/3394f69cdd6946c38e4b2998daf786d8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/730488298f7c4a1ab323a7f83ce2bd01, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/c889db32e61e45cca70149c3352ae11d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/44a417f5225c4775b79f0ea995828736, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/db8afb9daca94fd38e41addbe5f54d41, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ba3970e45d434237979ea44424c856ed, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/581e1388996f466fb317329cd215c32b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ce789affcbd34fe3aaf0b0d0d3736260, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9e3752eb5d60451cb829c17378ed6920, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/2efaf0a3318549599e9d2766e3c6d3be, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/adae205c968b415a9ea5feac4292b17a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8187dd7671014c98a434e984d259f8cd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9b4cd55b70c54143b8868061b42d3652, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/bec0c43acb6f42f09a6f14fb53e57e08, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/eb5e22d6028c4053a48f7c0be03884c9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/579cc046ed894233b255c22e4756dc75, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/e1794f4c05aa418b8b0800186a13b1ee, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/b52723a177e64c7c94755d512c9e27c2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/5558a632322a4e6393ece52519a26b15, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/088e4a76e7f840cc8b4cf7961c1245b9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/4fad471e68134f0dadecdc1e7f65e994, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9ecb0dd2e0e146d6b4c68db725f1d9d9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/08c933df7d424cb1b0d6e3d9861cd4d4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/66c7c33d057b4e1ea53b2257fa1957e6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/581900d285084c0bb7014558981eb2ba, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9adf08cc22dc4c51acfde8b090169017, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8bbbe2ae88934210b220318b1f93d71a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/a60cd41d5d5f421fa88ccabee64c8606, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/cee876bb351941d88fd1164b3a6db6e8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/5a485a7c7b1047a58854c1e61dc2b983, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ff03fb83a558409ab69765720ec22396, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/719e9000596249168d919b778c0741f9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/cb9cf11934ba43c494bc1a4c680ae801, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/e11ad43d0a8f47789bbc48012329ae6b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/7cf5124f5a5d4dcea966f8854f7b597b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/f5c79682e49e4926b559910722adcb66, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/290b908d6f1344a5af4bf89eef3bf570, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/711560989f0447d8b98944d7e86be49e] to archive 2023-04-03 05:56:15,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:56:15,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9cea537836f14288a89cbd2207ee44d0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9cea537836f14288a89cbd2207ee44d0 2023-04-03 05:56:15,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/3394f69cdd6946c38e4b2998daf786d8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/3394f69cdd6946c38e4b2998daf786d8 2023-04-03 05:56:15,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/730488298f7c4a1ab323a7f83ce2bd01 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/730488298f7c4a1ab323a7f83ce2bd01 2023-04-03 05:56:15,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/c889db32e61e45cca70149c3352ae11d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/c889db32e61e45cca70149c3352ae11d 2023-04-03 05:56:15,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/44a417f5225c4775b79f0ea995828736 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/44a417f5225c4775b79f0ea995828736 2023-04-03 05:56:15,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/db8afb9daca94fd38e41addbe5f54d41 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/db8afb9daca94fd38e41addbe5f54d41 2023-04-03 05:56:15,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ba3970e45d434237979ea44424c856ed to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ba3970e45d434237979ea44424c856ed 2023-04-03 05:56:15,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/581e1388996f466fb317329cd215c32b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/581e1388996f466fb317329cd215c32b 2023-04-03 05:56:15,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ce789affcbd34fe3aaf0b0d0d3736260 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ce789affcbd34fe3aaf0b0d0d3736260 2023-04-03 05:56:15,316 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9e3752eb5d60451cb829c17378ed6920 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9e3752eb5d60451cb829c17378ed6920 2023-04-03 05:56:15,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/2efaf0a3318549599e9d2766e3c6d3be to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/2efaf0a3318549599e9d2766e3c6d3be 2023-04-03 05:56:15,318 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/adae205c968b415a9ea5feac4292b17a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/adae205c968b415a9ea5feac4292b17a 2023-04-03 05:56:15,319 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8187dd7671014c98a434e984d259f8cd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8187dd7671014c98a434e984d259f8cd 2023-04-03 05:56:15,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9b4cd55b70c54143b8868061b42d3652 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9b4cd55b70c54143b8868061b42d3652 2023-04-03 05:56:15,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/bec0c43acb6f42f09a6f14fb53e57e08 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/bec0c43acb6f42f09a6f14fb53e57e08 2023-04-03 05:56:15,322 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/eb5e22d6028c4053a48f7c0be03884c9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/eb5e22d6028c4053a48f7c0be03884c9 2023-04-03 05:56:15,323 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/579cc046ed894233b255c22e4756dc75 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/579cc046ed894233b255c22e4756dc75 2023-04-03 05:56:15,324 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/e1794f4c05aa418b8b0800186a13b1ee to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/e1794f4c05aa418b8b0800186a13b1ee 2023-04-03 05:56:15,325 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/b52723a177e64c7c94755d512c9e27c2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/b52723a177e64c7c94755d512c9e27c2 2023-04-03 05:56:15,326 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/5558a632322a4e6393ece52519a26b15 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/5558a632322a4e6393ece52519a26b15 2023-04-03 05:56:15,327 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/088e4a76e7f840cc8b4cf7961c1245b9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/088e4a76e7f840cc8b4cf7961c1245b9 2023-04-03 05:56:15,327 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/4fad471e68134f0dadecdc1e7f65e994 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/4fad471e68134f0dadecdc1e7f65e994 2023-04-03 05:56:15,328 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9ecb0dd2e0e146d6b4c68db725f1d9d9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9ecb0dd2e0e146d6b4c68db725f1d9d9 2023-04-03 05:56:15,329 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/08c933df7d424cb1b0d6e3d9861cd4d4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/08c933df7d424cb1b0d6e3d9861cd4d4 2023-04-03 05:56:15,330 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/66c7c33d057b4e1ea53b2257fa1957e6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/66c7c33d057b4e1ea53b2257fa1957e6 2023-04-03 05:56:15,330 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/581900d285084c0bb7014558981eb2ba to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/581900d285084c0bb7014558981eb2ba 2023-04-03 05:56:15,331 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9adf08cc22dc4c51acfde8b090169017 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9adf08cc22dc4c51acfde8b090169017 2023-04-03 05:56:15,332 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8bbbe2ae88934210b220318b1f93d71a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8bbbe2ae88934210b220318b1f93d71a 2023-04-03 05:56:15,333 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/a60cd41d5d5f421fa88ccabee64c8606 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/a60cd41d5d5f421fa88ccabee64c8606 2023-04-03 05:56:15,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/cee876bb351941d88fd1164b3a6db6e8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/cee876bb351941d88fd1164b3a6db6e8 2023-04-03 05:56:15,335 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/5a485a7c7b1047a58854c1e61dc2b983 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/5a485a7c7b1047a58854c1e61dc2b983 2023-04-03 05:56:15,336 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ff03fb83a558409ab69765720ec22396 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/ff03fb83a558409ab69765720ec22396 2023-04-03 05:56:15,337 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/719e9000596249168d919b778c0741f9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/719e9000596249168d919b778c0741f9 2023-04-03 05:56:15,338 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/cb9cf11934ba43c494bc1a4c680ae801 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/cb9cf11934ba43c494bc1a4c680ae801 2023-04-03 05:56:15,339 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/e11ad43d0a8f47789bbc48012329ae6b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/e11ad43d0a8f47789bbc48012329ae6b 2023-04-03 05:56:15,340 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/7cf5124f5a5d4dcea966f8854f7b597b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/7cf5124f5a5d4dcea966f8854f7b597b 2023-04-03 05:56:15,341 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/f5c79682e49e4926b559910722adcb66 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/f5c79682e49e4926b559910722adcb66 2023-04-03 05:56:15,342 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/290b908d6f1344a5af4bf89eef3bf570 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/290b908d6f1344a5af4bf89eef3bf570 2023-04-03 05:56:15,342 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/711560989f0447d8b98944d7e86be49e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/711560989f0447d8b98944d7e86be49e 2023-04-03 05:56:15,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/a2fb9ad8a04449f5a43f02f3bdee0d19, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b365c783d3fe4702b5254b51b6bbf9ee, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b32e89ffe9d64c5191ad9bf896098363, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1b0bbc6ab44e4d14b73186a83a2eb99e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/7d3f8a709c584333b5a40a921eeb92d3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/d8651c68a5ca4cef8afee35bbcf914ac, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/cba42305e1714bb3b454ef46803640b0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/133978a2bc104b20abf28566e8625619, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/db644e96d71e4c42a118a369cf034d3d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/470c5eed07de4770970e7e68db7b948f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/a79aae389e6243468386fe1ff0174b73, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/74d9c4409fb44344948167e5f4291385, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/741c07e32db046c1935b667a7b7e9e21, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/2d4e906f8ffe41c8a0b6c1f36a24edcc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/0b8007e30fec4dcd89145972838ce08d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/c38b1b42fc414137b82612ee0b0dd0eb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/572c354bb0574d8482f37d9532886e38, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/60d32876aa6a41e4822ebddf04be2324, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/247a0c05793142d7bf89a4f34afb0a38, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/25be44160438474782ad1131dc4f64fb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/24bba2960e3c4dacaa3121d36f118a3c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/727398b17b30420f81ce8e5a4d4cd3e7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/bcd6b6608cda4eae9aa426567f95b825, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/526450d71cfa4b99b1a48a4f93a6a457, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b6511c99251841eda4c0fed23f50fe4f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/352655293e174972a5dd3572e98c8129, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/ccd83e653a3d40f88dd0f28b068acd52, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/fa0566e151064830b9413635e8f0d2de, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/58dfb604dd8d4b039474134ca56ba864, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f1b28944c64e48d68363c10863df4cca, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/6b3bd689de4b4a0ba228d8d94e8ef178, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1041926654ce4e8dacc56e93220fd85a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/73170c6d919f425c898fb718ec681a0b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/fe94a23986d8473d9ec509204502ce50, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/7ed32784b7df4610ab054230a043ec15, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/423bb3fdb0bd4de9b2863c4f8e4b1989, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/66af248aa1f248dd807bbfedf6e3dce3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f8cdbda990654f15bdf9bdab3e7766db, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/eb716506efed4bbcb7200ff00375fa93] to archive 2023-04-03 05:56:15,346 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:56:15,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/a2fb9ad8a04449f5a43f02f3bdee0d19 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/a2fb9ad8a04449f5a43f02f3bdee0d19 2023-04-03 05:56:15,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b365c783d3fe4702b5254b51b6bbf9ee to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b365c783d3fe4702b5254b51b6bbf9ee 2023-04-03 05:56:15,349 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b32e89ffe9d64c5191ad9bf896098363 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b32e89ffe9d64c5191ad9bf896098363 2023-04-03 05:56:15,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1b0bbc6ab44e4d14b73186a83a2eb99e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1b0bbc6ab44e4d14b73186a83a2eb99e 2023-04-03 05:56:15,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/7d3f8a709c584333b5a40a921eeb92d3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/7d3f8a709c584333b5a40a921eeb92d3 2023-04-03 05:56:15,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/d8651c68a5ca4cef8afee35bbcf914ac to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/d8651c68a5ca4cef8afee35bbcf914ac 2023-04-03 05:56:15,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/cba42305e1714bb3b454ef46803640b0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/cba42305e1714bb3b454ef46803640b0 2023-04-03 05:56:15,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/133978a2bc104b20abf28566e8625619 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/133978a2bc104b20abf28566e8625619 2023-04-03 05:56:15,354 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/db644e96d71e4c42a118a369cf034d3d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/db644e96d71e4c42a118a369cf034d3d 2023-04-03 05:56:15,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/470c5eed07de4770970e7e68db7b948f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/470c5eed07de4770970e7e68db7b948f 2023-04-03 05:56:15,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/a79aae389e6243468386fe1ff0174b73 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/a79aae389e6243468386fe1ff0174b73 2023-04-03 05:56:15,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/74d9c4409fb44344948167e5f4291385 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/74d9c4409fb44344948167e5f4291385 2023-04-03 05:56:15,357 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/741c07e32db046c1935b667a7b7e9e21 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/741c07e32db046c1935b667a7b7e9e21 2023-04-03 05:56:15,358 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/2d4e906f8ffe41c8a0b6c1f36a24edcc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/2d4e906f8ffe41c8a0b6c1f36a24edcc 2023-04-03 05:56:15,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/0b8007e30fec4dcd89145972838ce08d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/0b8007e30fec4dcd89145972838ce08d 2023-04-03 05:56:15,359 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/c38b1b42fc414137b82612ee0b0dd0eb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/c38b1b42fc414137b82612ee0b0dd0eb 2023-04-03 05:56:15,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/572c354bb0574d8482f37d9532886e38 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/572c354bb0574d8482f37d9532886e38 2023-04-03 05:56:15,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/60d32876aa6a41e4822ebddf04be2324 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/60d32876aa6a41e4822ebddf04be2324 2023-04-03 05:56:15,362 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/247a0c05793142d7bf89a4f34afb0a38 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/247a0c05793142d7bf89a4f34afb0a38 2023-04-03 05:56:15,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/25be44160438474782ad1131dc4f64fb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/25be44160438474782ad1131dc4f64fb 2023-04-03 05:56:15,364 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/24bba2960e3c4dacaa3121d36f118a3c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/24bba2960e3c4dacaa3121d36f118a3c 2023-04-03 05:56:15,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/727398b17b30420f81ce8e5a4d4cd3e7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/727398b17b30420f81ce8e5a4d4cd3e7 2023-04-03 05:56:15,367 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/bcd6b6608cda4eae9aa426567f95b825 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/bcd6b6608cda4eae9aa426567f95b825 2023-04-03 05:56:15,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/526450d71cfa4b99b1a48a4f93a6a457 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/526450d71cfa4b99b1a48a4f93a6a457 2023-04-03 05:56:15,369 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b6511c99251841eda4c0fed23f50fe4f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/b6511c99251841eda4c0fed23f50fe4f 2023-04-03 05:56:15,370 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/352655293e174972a5dd3572e98c8129 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/352655293e174972a5dd3572e98c8129 2023-04-03 05:56:15,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/ccd83e653a3d40f88dd0f28b068acd52 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/ccd83e653a3d40f88dd0f28b068acd52 2023-04-03 05:56:15,372 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/fa0566e151064830b9413635e8f0d2de to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/fa0566e151064830b9413635e8f0d2de 2023-04-03 05:56:15,373 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/58dfb604dd8d4b039474134ca56ba864 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/58dfb604dd8d4b039474134ca56ba864 2023-04-03 05:56:15,374 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f1b28944c64e48d68363c10863df4cca to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f1b28944c64e48d68363c10863df4cca 2023-04-03 05:56:15,374 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/6b3bd689de4b4a0ba228d8d94e8ef178 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/6b3bd689de4b4a0ba228d8d94e8ef178 2023-04-03 05:56:15,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1041926654ce4e8dacc56e93220fd85a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1041926654ce4e8dacc56e93220fd85a 2023-04-03 05:56:15,376 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/73170c6d919f425c898fb718ec681a0b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/73170c6d919f425c898fb718ec681a0b 2023-04-03 05:56:15,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/fe94a23986d8473d9ec509204502ce50 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/fe94a23986d8473d9ec509204502ce50 2023-04-03 05:56:15,378 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/7ed32784b7df4610ab054230a043ec15 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/7ed32784b7df4610ab054230a043ec15 2023-04-03 05:56:15,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/423bb3fdb0bd4de9b2863c4f8e4b1989 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/423bb3fdb0bd4de9b2863c4f8e4b1989 2023-04-03 05:56:15,381 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/66af248aa1f248dd807bbfedf6e3dce3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/66af248aa1f248dd807bbfedf6e3dce3 2023-04-03 05:56:15,382 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f8cdbda990654f15bdf9bdab3e7766db to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f8cdbda990654f15bdf9bdab3e7766db 2023-04-03 05:56:15,383 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/eb716506efed4bbcb7200ff00375fa93 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/eb716506efed4bbcb7200ff00375fa93 2023-04-03 05:56:15,385 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/aa7f32cf79de4496af21c86790e76bc5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/2e65cae376d84ffcb2ab8f8e471b1ed8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/312d7ba753034b968e50d8da1e67d9ff, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/58e3e006dc8445659f30e6548c63ab74, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/e58cacc76b0b4a49b604efe904b79ea9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/26b2f51e902a4cbc8088d53f06f96bf9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/1f8804eb0e6a49c69208812d3ea7eb97, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cec18d6dbd6a4683ab29db90ed4457ba, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/6953e1f2804e4cff83cd0150a76d8b84, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/23978a2bbe8f44d5b273c9d3fca79605, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/174a5951d4594d33a46b30a624645466, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/00c75a671776409fa4d336dfcf222768, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/0232b6e8d9ea4b7eacfd6a67f3c1cceb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/eefef8ca7d4649a39234cfd435e20c7f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/b3e9f04abf274013a5e27c9f5ee7eae4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cebf34cff22e4e38b3a545c13e19cff3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/01fcbb95a1c5493fb14750a6ecf7cb27, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/3049535ef1844b8e9dec3f7fcaa11113, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/41a5c3d05e58452b9ce53cc5ccb964df, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/a66a03e17b424d6ebba1904e1bf5569c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/da81b1b14f5543c0b47a8503a8ccbcf5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/edab102d7470499686ac5924ffae8dff, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/29bd891365d74dd8b6bc1f5023d7160b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/eec06d3e96b140ebb8c171a2c15a81c0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/d5194b0cbca14cb49af40de7c318e849, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/51585e4b030b4c68af85c5aaf4cd4267, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/d52afe76a6c24d4f8aceb96fa4c31da0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/721e671b8afd4adea7abfa7a2bb7c2f3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/7e26d10ca08b42f0acd5b70e64965c9f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/dcba649505c945ceafedb7edf3223d7c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/069e0067168242aca4ca2d31a08584d8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/efcf7837650249b382c3be075bc3a92c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/f5371569c10a4914942a610c280566fc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/7ad068c39e8d452da18f4d5dc3c42d93, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/c241f5fda9dc4cebabd8f9df40c2ec8d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/c3782cf296784415aeb2be460940b8d0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/4026d200b76e4d3cbc957d88e8c968b7] to archive 2023-04-03 05:56:15,386 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:56:15,387 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/aa7f32cf79de4496af21c86790e76bc5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/aa7f32cf79de4496af21c86790e76bc5 2023-04-03 05:56:15,388 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/2e65cae376d84ffcb2ab8f8e471b1ed8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/2e65cae376d84ffcb2ab8f8e471b1ed8 2023-04-03 05:56:15,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/312d7ba753034b968e50d8da1e67d9ff to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/312d7ba753034b968e50d8da1e67d9ff 2023-04-03 05:56:15,389 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/58e3e006dc8445659f30e6548c63ab74 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/58e3e006dc8445659f30e6548c63ab74 2023-04-03 05:56:15,390 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/e58cacc76b0b4a49b604efe904b79ea9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/e58cacc76b0b4a49b604efe904b79ea9 2023-04-03 05:56:15,391 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/26b2f51e902a4cbc8088d53f06f96bf9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/26b2f51e902a4cbc8088d53f06f96bf9 2023-04-03 05:56:15,392 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/1f8804eb0e6a49c69208812d3ea7eb97 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/1f8804eb0e6a49c69208812d3ea7eb97 2023-04-03 05:56:15,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cec18d6dbd6a4683ab29db90ed4457ba to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cec18d6dbd6a4683ab29db90ed4457ba 2023-04-03 05:56:15,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/6953e1f2804e4cff83cd0150a76d8b84 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/6953e1f2804e4cff83cd0150a76d8b84 2023-04-03 05:56:15,394 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/23978a2bbe8f44d5b273c9d3fca79605 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/23978a2bbe8f44d5b273c9d3fca79605 2023-04-03 05:56:15,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/174a5951d4594d33a46b30a624645466 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/174a5951d4594d33a46b30a624645466 2023-04-03 05:56:15,396 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/00c75a671776409fa4d336dfcf222768 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/00c75a671776409fa4d336dfcf222768 2023-04-03 05:56:15,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 05:56:15,397 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/0232b6e8d9ea4b7eacfd6a67f3c1cceb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/0232b6e8d9ea4b7eacfd6a67f3c1cceb 2023-04-03 05:56:15,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/eefef8ca7d4649a39234cfd435e20c7f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/eefef8ca7d4649a39234cfd435e20c7f 2023-04-03 05:56:15,398 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/b3e9f04abf274013a5e27c9f5ee7eae4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/b3e9f04abf274013a5e27c9f5ee7eae4 2023-04-03 05:56:15,399 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cebf34cff22e4e38b3a545c13e19cff3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cebf34cff22e4e38b3a545c13e19cff3 2023-04-03 05:56:15,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/01fcbb95a1c5493fb14750a6ecf7cb27 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/01fcbb95a1c5493fb14750a6ecf7cb27 2023-04-03 05:56:15,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/3049535ef1844b8e9dec3f7fcaa11113 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/3049535ef1844b8e9dec3f7fcaa11113 2023-04-03 05:56:15,401 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/41a5c3d05e58452b9ce53cc5ccb964df to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/41a5c3d05e58452b9ce53cc5ccb964df 2023-04-03 05:56:15,402 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/a66a03e17b424d6ebba1904e1bf5569c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/a66a03e17b424d6ebba1904e1bf5569c 2023-04-03 05:56:15,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/da81b1b14f5543c0b47a8503a8ccbcf5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/da81b1b14f5543c0b47a8503a8ccbcf5 2023-04-03 05:56:15,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/edab102d7470499686ac5924ffae8dff to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/edab102d7470499686ac5924ffae8dff 2023-04-03 05:56:15,404 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/29bd891365d74dd8b6bc1f5023d7160b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/29bd891365d74dd8b6bc1f5023d7160b 2023-04-03 05:56:15,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/eec06d3e96b140ebb8c171a2c15a81c0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/eec06d3e96b140ebb8c171a2c15a81c0 2023-04-03 05:56:15,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/d5194b0cbca14cb49af40de7c318e849 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/d5194b0cbca14cb49af40de7c318e849 2023-04-03 05:56:15,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/51585e4b030b4c68af85c5aaf4cd4267 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/51585e4b030b4c68af85c5aaf4cd4267 2023-04-03 05:56:15,407 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/d52afe76a6c24d4f8aceb96fa4c31da0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/d52afe76a6c24d4f8aceb96fa4c31da0 2023-04-03 05:56:15,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/721e671b8afd4adea7abfa7a2bb7c2f3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/721e671b8afd4adea7abfa7a2bb7c2f3 2023-04-03 05:56:15,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/7e26d10ca08b42f0acd5b70e64965c9f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/7e26d10ca08b42f0acd5b70e64965c9f 2023-04-03 05:56:15,409 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/dcba649505c945ceafedb7edf3223d7c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/dcba649505c945ceafedb7edf3223d7c 2023-04-03 05:56:15,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/069e0067168242aca4ca2d31a08584d8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/069e0067168242aca4ca2d31a08584d8 2023-04-03 05:56:15,410 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/efcf7837650249b382c3be075bc3a92c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/efcf7837650249b382c3be075bc3a92c 2023-04-03 05:56:15,411 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/f5371569c10a4914942a610c280566fc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/f5371569c10a4914942a610c280566fc 2023-04-03 05:56:15,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/7ad068c39e8d452da18f4d5dc3c42d93 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/7ad068c39e8d452da18f4d5dc3c42d93 2023-04-03 05:56:15,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/c241f5fda9dc4cebabd8f9df40c2ec8d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/c241f5fda9dc4cebabd8f9df40c2ec8d 2023-04-03 05:56:15,416 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/c3782cf296784415aeb2be460940b8d0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/c3782cf296784415aeb2be460940b8d0 2023-04-03 05:56:15,416 DEBUG [StoreCloser-TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/4026d200b76e4d3cbc957d88e8c968b7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/4026d200b76e4d3cbc957d88e8c968b7 2023-04-03 05:56:15,420 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/recovered.edits/700.seqid, newMaxSeqId=700, maxSeqId=1 2023-04-03 05:56:15,422 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a. 2023-04-03 05:56:15,422 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5d777b61781e184e82ec86ca4072f25a: 2023-04-03 05:56:15,423 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:56:15,424 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=5d777b61781e184e82ec86ca4072f25a, regionState=CLOSED 2023-04-03 05:56:15,424 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501375424"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501375424"}]},"ts":"1680501375424"} 2023-04-03 05:56:15,426 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-03 05:56:15,426 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 5d777b61781e184e82ec86ca4072f25a, server=jenkins-hbase4.apache.org,37397,1680501214665 in 1.1260 sec 2023-04-03 05:56:15,427 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-03 05:56:15,427 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5d777b61781e184e82ec86ca4072f25a, UNASSIGN in 1.1300 sec 2023-04-03 05:56:15,427 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501375427"}]},"ts":"1680501375427"} 2023-04-03 05:56:15,428 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 05:56:15,429 INFO [PEWorker-4] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 05:56:15,430 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1400 sec 2023-04-03 05:56:16,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-03 05:56:16,397 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-03 05:56:16,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-03 05:56:16,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:16,400 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:16,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-03 05:56:16,401 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:16,405 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:56:16,407 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/recovered.edits] 2023-04-03 05:56:16,409 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8f4f6c8e217d433c8e3f9557d8a357b5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/8f4f6c8e217d433c8e3f9557d8a357b5 2023-04-03 05:56:16,410 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9685d408dbbc42acbad623750a1541dd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/9685d408dbbc42acbad623750a1541dd 2023-04-03 05:56:16,411 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/d53e714e40f04498b6dc8ead1e1e314f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/A/d53e714e40f04498b6dc8ead1e1e314f 2023-04-03 05:56:16,413 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1d927709ded846ffbc3ea1ac7a1d76f6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/1d927709ded846ffbc3ea1ac7a1d76f6 2023-04-03 05:56:16,414 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/d8e6d0be91b54c8a8720a5f38e251bf6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/d8e6d0be91b54c8a8720a5f38e251bf6 2023-04-03 05:56:16,415 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f04c5484dbf44b29bec07b9cf7070f1a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/B/f04c5484dbf44b29bec07b9cf7070f1a 2023-04-03 05:56:16,417 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cfc40488d8404cb5b1a0490cff1f754f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/cfc40488d8404cb5b1a0490cff1f754f 2023-04-03 05:56:16,418 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/dcee3a93424f4a6da11cfecf313a3a20 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/dcee3a93424f4a6da11cfecf313a3a20 2023-04-03 05:56:16,418 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/edaa1e2f03bd41ac924df0c34ea214e4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/C/edaa1e2f03bd41ac924df0c34ea214e4 2023-04-03 05:56:16,420 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/recovered.edits/700.seqid to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a/recovered.edits/700.seqid 2023-04-03 05:56:16,421 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/5d777b61781e184e82ec86ca4072f25a 2023-04-03 05:56:16,421 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 05:56:16,422 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:16,423 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 05:56:16,425 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 05:56:16,425 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:16,425 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 05:56:16,426 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680501376426"}]},"ts":"9223372036854775807"} 2023-04-03 05:56:16,427 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 05:56:16,427 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5d777b61781e184e82ec86ca4072f25a, NAME => 'TestAcidGuarantees,,1680501343539.5d777b61781e184e82ec86ca4072f25a.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 05:56:16,427 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 05:56:16,427 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680501376427"}]},"ts":"9223372036854775807"} 2023-04-03 05:56:16,428 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 05:56:16,430 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:16,431 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 32 msec 2023-04-03 05:56:16,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-03 05:56:16,502 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-03 05:56:16,517 INFO [Listener at localhost/37275] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 230), OpenFileDescriptor=544 (was 541) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=400 (was 438), ProcessCount=169 (was 169), AvailableMemoryMB=3785 (was 3353) - AvailableMemoryMB LEAK? - 2023-04-03 05:56:16,529 INFO [Listener at localhost/37275] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=544, MaxFileDescriptor=60000, SystemLoadAverage=400, ProcessCount=169, AvailableMemoryMB=3784 2023-04-03 05:56:16,532 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 05:56:16,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 05:56:16,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:16,536 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-03 05:56:16,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-03 05:56:16,536 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-03 05:56:16,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 05:56:16,537 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:16,538 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6 empty. 2023-04-03 05:56:16,538 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:16,538 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 05:56:16,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 05:56:16,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 05:56:16,948 DEBUG [PEWorker-5] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 05:56:16,950 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 67f8cb6d380f875bf3bd14331600fad6, NAME => 'TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp 2023-04-03 05:56:17,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 05:56:17,355 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:56:17,355 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 67f8cb6d380f875bf3bd14331600fad6, disabling compactions & flushes 2023-04-03 05:56:17,355 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:17,355 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:17,355 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. after waiting 0 ms 2023-04-03 05:56:17,355 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:17,355 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:17,355 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:17,358 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-03 05:56:17,359 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501377359"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501377359"}]},"ts":"1680501377359"} 2023-04-03 05:56:17,360 INFO [PEWorker-5] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-03 05:56:17,360 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-03 05:56:17,360 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501377360"}]},"ts":"1680501377360"} 2023-04-03 05:56:17,361 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-03 05:56:17,365 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=67f8cb6d380f875bf3bd14331600fad6, ASSIGN}] 2023-04-03 05:56:17,367 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=67f8cb6d380f875bf3bd14331600fad6, ASSIGN 2023-04-03 05:56:17,367 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=67f8cb6d380f875bf3bd14331600fad6, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,37397,1680501214665; forceNewPlan=false, retain=false 2023-04-03 05:56:17,518 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=67f8cb6d380f875bf3bd14331600fad6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:17,518 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501377518"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501377518"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501377518"}]},"ts":"1680501377518"} 2023-04-03 05:56:17,520 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:56:17,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 05:56:17,674 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:17,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 67f8cb6d380f875bf3bd14331600fad6, NAME => 'TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.', STARTKEY => '', ENDKEY => ''} 2023-04-03 05:56:17,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:17,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:56:17,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:17,674 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:17,676 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:17,677 DEBUG [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A 2023-04-03 05:56:17,677 DEBUG [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A 2023-04-03 05:56:17,677 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:56:17,677 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 67f8cb6d380f875bf3bd14331600fad6 columnFamilyName A 2023-04-03 05:56:17,678 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.HStore(310): Store=67f8cb6d380f875bf3bd14331600fad6/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:56:17,678 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:17,679 DEBUG [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B 2023-04-03 05:56:17,679 DEBUG [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B 2023-04-03 05:56:17,679 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:56:17,679 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 67f8cb6d380f875bf3bd14331600fad6 columnFamilyName B 2023-04-03 05:56:17,680 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.HStore(310): Store=67f8cb6d380f875bf3bd14331600fad6/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:56:17,680 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:17,680 DEBUG [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C 2023-04-03 05:56:17,680 DEBUG [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C 2023-04-03 05:56:17,681 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:56:17,681 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 67f8cb6d380f875bf3bd14331600fad6 columnFamilyName C 2023-04-03 05:56:17,681 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.HStore(310): Store=67f8cb6d380f875bf3bd14331600fad6/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:56:17,681 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:17,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:17,682 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:17,683 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 05:56:17,684 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:17,685 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-03 05:56:17,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 67f8cb6d380f875bf3bd14331600fad6; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68228854, jitterRate=0.016689151525497437}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 05:56:17,686 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:17,686 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., pid=56, masterSystemTime=1680501377671 2023-04-03 05:56:17,687 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:17,687 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:17,687 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=67f8cb6d380f875bf3bd14331600fad6, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:17,688 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501377687"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680501377687"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680501377687"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680501377687"}]},"ts":"1680501377687"} 2023-04-03 05:56:17,689 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-03 05:56:17,689 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 in 168 msec 2023-04-03 05:56:17,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-03 05:56:17,690 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=67f8cb6d380f875bf3bd14331600fad6, ASSIGN in 324 msec 2023-04-03 05:56:17,691 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-03 05:56:17,691 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501377691"}]},"ts":"1680501377691"} 2023-04-03 05:56:17,691 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-03 05:56:17,693 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-03 05:56:17,694 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-04-03 05:56:18,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-03 05:56:18,641 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-03 05:56:18,642 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x54cb8b37 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:56:18,646 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5cb37c16, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:56:18,647 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:56:18,648 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39348, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:56:18,649 DEBUG [Listener at localhost/37275] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-03 05:56:18,651 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:59046, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-03 05:56:18,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-03 05:56:18,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-03 05:56:18,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:19,061 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 05:56:19,062 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-03 05:56:19,063 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-03 05:56:19,064 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-03 05:56:19,067 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=67f8cb6d380f875bf3bd14331600fad6, REOPEN/MOVE}] 2023-04-03 05:56:19,067 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=67f8cb6d380f875bf3bd14331600fad6, REOPEN/MOVE 2023-04-03 05:56:19,067 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=67f8cb6d380f875bf3bd14331600fad6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,068 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501379067"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501379067"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501379067"}]},"ts":"1680501379067"} 2023-04-03 05:56:19,068 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:56:19,220 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 67f8cb6d380f875bf3bd14331600fad6, disabling compactions & flushes 2023-04-03 05:56:19,221 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. after waiting 0 ms 2023-04-03 05:56:19,221 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,223 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-03 05:56:19,224 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,224 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:19,224 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 67f8cb6d380f875bf3bd14331600fad6 to self. 2023-04-03 05:56:19,225 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,225 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=67f8cb6d380f875bf3bd14331600fad6, regionState=CLOSED 2023-04-03 05:56:19,225 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501379225"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501379225"}]},"ts":"1680501379225"} 2023-04-03 05:56:19,227 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-03 05:56:19,227 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 in 158 msec 2023-04-03 05:56:19,228 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=67f8cb6d380f875bf3bd14331600fad6, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,37397,1680501214665; forceNewPlan=false, retain=true 2023-04-03 05:56:19,378 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=67f8cb6d380f875bf3bd14331600fad6, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,378 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501379378"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501379378"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501379378"}]},"ts":"1680501379378"} 2023-04-03 05:56:19,379 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:56:19,534 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,534 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 67f8cb6d380f875bf3bd14331600fad6, NAME => 'TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.', STARTKEY => '', ENDKEY => ''} 2023-04-03 05:56:19,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-03 05:56:19,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,535 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,536 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,537 DEBUG [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A 2023-04-03 05:56:19,537 DEBUG [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A 2023-04-03 05:56:19,537 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:56:19,537 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 67f8cb6d380f875bf3bd14331600fad6 columnFamilyName A 2023-04-03 05:56:19,539 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.HStore(310): Store=67f8cb6d380f875bf3bd14331600fad6/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:56:19,539 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,540 DEBUG [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B 2023-04-03 05:56:19,540 DEBUG [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B 2023-04-03 05:56:19,540 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:56:19,541 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 67f8cb6d380f875bf3bd14331600fad6 columnFamilyName B 2023-04-03 05:56:19,541 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.HStore(310): Store=67f8cb6d380f875bf3bd14331600fad6/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:56:19,541 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,542 DEBUG [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C 2023-04-03 05:56:19,542 DEBUG [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C 2023-04-03 05:56:19,542 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-03 05:56:19,542 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 67f8cb6d380f875bf3bd14331600fad6 columnFamilyName C 2023-04-03 05:56:19,543 INFO [StoreOpener-67f8cb6d380f875bf3bd14331600fad6-1] regionserver.HStore(310): Store=67f8cb6d380f875bf3bd14331600fad6/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-03 05:56:19,543 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,544 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,546 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,548 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-03 05:56:19,549 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,550 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 67f8cb6d380f875bf3bd14331600fad6; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=63660946, jitterRate=-0.05137798190116882}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-03 05:56:19,550 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:19,550 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., pid=61, masterSystemTime=1680501379531 2023-04-03 05:56:19,552 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,552 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,552 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=67f8cb6d380f875bf3bd14331600fad6, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,552 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501379552"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680501379552"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680501379552"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680501379552"}]},"ts":"1680501379552"} 2023-04-03 05:56:19,554 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-03 05:56:19,554 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 in 174 msec 2023-04-03 05:56:19,555 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-03 05:56:19,555 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=67f8cb6d380f875bf3bd14331600fad6, REOPEN/MOVE in 488 msec 2023-04-03 05:56:19,556 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-03 05:56:19,557 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 491 msec 2023-04-03 05:56:19,558 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 905 msec 2023-04-03 05:56:19,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-03 05:56:19,559 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x041dc777 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:56:19,564 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@c0766f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:56:19,564 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x089b6154 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:56:19,569 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6257294f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:56:19,569 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x01d207bb to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:56:19,573 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52104714, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:56:19,573 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x5bcc330a to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:56:19,579 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4a320817, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:56:19,579 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x23afc0f2 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:56:19,582 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@19c1e3ee, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:56:19,583 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x526f2221 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:56:19,588 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@44277511, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:56:19,588 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x17cf2e3f to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:56:19,595 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3c1a68cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:56:19,596 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x3420e54c to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:56:19,603 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a709b21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:56:19,603 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x26d7d0b5 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:56:19,608 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@156e4ba6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:56:19,608 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(139): Connect 0x601410a4 to 127.0.0.1:63744 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-03 05:56:19,616 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@785e366b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-03 05:56:19,628 DEBUG [hconnection-0x553f760e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:56:19,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:19,629 DEBUG [hconnection-0x1b451912-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:56:19,632 DEBUG [hconnection-0x548fb1c5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:56:19,632 DEBUG [hconnection-0x1dd3eb90-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:56:19,632 DEBUG [hconnection-0x742206cf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:56:19,633 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39388, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:56:19,636 DEBUG [hconnection-0x1978efdf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:56:19,637 DEBUG [hconnection-0x646dbd07-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:56:19,637 DEBUG [hconnection-0x31962f8b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:56:19,637 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39416, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:56:19,640 DEBUG [hconnection-0xce11fc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:56:19,640 DEBUG [hconnection-0x714d7cbd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-03 05:56:19,640 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39362, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:56:19,640 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39376, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:56:19,641 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39400, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:56:19,641 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39412, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:56:19,641 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39448, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:56:19,643 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39436, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:56:19,648 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39422, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:56:19,649 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39462, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-03 05:56:19,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:19,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:19,657 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:19,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:19,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:19,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:19,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,657 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,659 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:19,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:19,659 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:19,659 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:19,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,660 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,660 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:19,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,660 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:19,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:19,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,660 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:19,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:19,661 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:19,661 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:19,661 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:19,661 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:19,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:19,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:19,677 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501439673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:19,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501439674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501379671, max timestamp=1680501379676Num uniques 859; 2023-04-03 05:56:19,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501379671, max timestamp=1680501379676Num uniques 859; 2023-04-03 05:56:19,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:19,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501379671, max timestamp=1680501379676Num uniques 859; 2023-04-03 05:56:19,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:19,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:19,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:19,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:19,680 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:19,681 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:19,682 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,682 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:19,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:19,682 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,682 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:19,684 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eba32bf[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:19,684 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:19,684 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,685 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,685 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,685 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:19,685 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:19,688 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:19,688 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,688 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:19,688 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:19,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:19,691 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:19,691 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:19,691 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:19,691 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:19,691 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:19,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:19,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:19,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:19,693 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,695 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,698 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:19,698 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:19,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:19,699 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:19,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:19,699 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,699 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,699 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,699 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,699 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,699 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,699 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403fced55f9f46741339efa35fb20150ed1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403fced55f9f46741339efa35fb20150ed1 2023-04-03 05:56:19,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:19,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:19,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:19,699 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:19,705 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:19,705 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:19,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:19,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39388 deadline: 1680501439705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,708 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:19,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39448 deadline: 1680501439705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,709 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=15, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/b12c3337eeb14917a787387c4cddb0b7 2023-04-03 05:56:19,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:19,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39462 deadline: 1680501439708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,723 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/a0a8f78c142146809182a67a9514b480 2023-04-03 05:56:19,735 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/e1c81ec703884187b9e36b87a64859f9 2023-04-03 05:56:19,739 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/b12c3337eeb14917a787387c4cddb0b7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/b12c3337eeb14917a787387c4cddb0b7 2023-04-03 05:56:19,743 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/b12c3337eeb14917a787387c4cddb0b7, entries=150, sequenceid=15, filesize=25.1 K 2023-04-03 05:56:19,744 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/a0a8f78c142146809182a67a9514b480 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/a0a8f78c142146809182a67a9514b480 2023-04-03 05:56:19,747 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/a0a8f78c142146809182a67a9514b480, entries=150, sequenceid=15, filesize=11.6 K 2023-04-03 05:56:19,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/e1c81ec703884187b9e36b87a64859f9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e1c81ec703884187b9e36b87a64859f9 2023-04-03 05:56:19,751 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e1c81ec703884187b9e36b87a64859f9, entries=150, sequenceid=15, filesize=11.6 K 2023-04-03 05:56:19,752 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 67f8cb6d380f875bf3bd14331600fad6 in 93ms, sequenceid=15, compaction requested=false 2023-04-03 05:56:19,752 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:19,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:19,780 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-04-03 05:56:19,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:19,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:19,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:19,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:19,780 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:19,780 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:19,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:19,793 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501439790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:19,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501439790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,803 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304036c003761f67f41b89da0c021ab90f2d0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036c003761f67f41b89da0c021ab90f2d0 2023-04-03 05:56:19,805 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:19,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:19,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:19,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:19,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39388 deadline: 1680501439806, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:19,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39448 deadline: 1680501439809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:19,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39462 deadline: 1680501439811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:19,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:19,825 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:19,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:19,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:19,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:19,826 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,826 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,826 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/963f161354934dce83697a7d2b69bf8c 2023-04-03 05:56:19,828 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:19,828 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:19,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:19,829 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,830 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:19,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:19,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:19,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:19,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:19,839 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:19,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:19,840 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,840 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:19,840 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:19,840 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,841 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:19,841 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:19,841 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,841 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:19,841 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:19,847 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:19,847 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:19,847 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,847 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:19,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:19,849 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/68bd3cf85f474b6aa02e1d22475b2f89 2023-04-03 05:56:19,849 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:19,849 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@167fc09[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:19,850 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,850 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:19,850 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:19,851 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:19,852 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:19,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:19,852 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:19,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:19,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:19,853 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:19,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:19,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:19,853 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:19,853 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:19,856 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,856 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,856 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:19,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:19,858 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,859 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:19,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:19,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:19,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:19,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,863 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:19,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:19,864 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:19,864 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:19,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:19,864 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:19,864 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:19,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:19,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501439894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,904 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:19,904 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501439901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,964 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:19,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:19,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:19,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:19,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:19,968 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:19,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:19,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:19,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:19,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,968 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:19,970 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:19,970 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,970 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:19,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:19,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:19,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:19,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:19,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:19,971 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:19,971 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:19,973 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:19,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59816449[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:19,974 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:19,975 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,976 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,976 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:19,976 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:19,976 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,976 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:19,976 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:19,978 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:19,978 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:19,978 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:19,982 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:19,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:19,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:19,985 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:19,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:19,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:19,985 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:19,985 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:19,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:19,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,986 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:19,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:19,987 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,988 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,990 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:19,990 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:19,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:19,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:19,990 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,990 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,990 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:19,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:19,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:19,990 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:19,991 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:19,991 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,991 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:19,991 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:19,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:19,991 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:19,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:19,991 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:20,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39388 deadline: 1680501440010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,015 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39462 deadline: 1680501440014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39448 deadline: 1680501440016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,091 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:20,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:20,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:20,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:20,096 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:20,096 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:20,096 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:20,096 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:20,097 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:20,097 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,097 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,098 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:20,098 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:20,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:20,099 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,099 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,101 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:20,101 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:20,101 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:20,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:20,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:20,101 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:20,105 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:20,105 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:20,105 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:20,105 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:20,105 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:20,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501440104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,106 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,106 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:20,106 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:20,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501440105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,108 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:20,109 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,109 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:20,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:20,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6022791b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:20,111 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:20,111 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,113 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,113 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:20,113 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,113 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:20,113 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:20,113 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:20,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:20,115 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,115 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:20,116 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:20,118 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:20,118 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:20,118 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:20,118 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:20,118 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:20,119 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:20,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:20,119 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:20,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:20,120 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,122 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:20,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:20,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:20,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:20,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:20,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,125 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:20,125 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:20,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:20,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:20,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:20,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:20,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:20,225 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:20,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:20,227 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:20,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:20,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:20,229 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:20,229 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:20,229 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:20,229 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:20,229 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,229 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,230 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:20,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:20,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:20,231 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,231 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:20,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:20,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:20,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:20,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:20,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:20,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:20,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:20,232 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:20,232 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:20,232 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:20,233 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:20,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:20,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:20,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:20,233 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:20,234 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,234 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:20,234 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,234 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fc2806a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:20,236 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:20,236 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,241 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,241 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:20,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:20,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:20,241 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:20,242 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,242 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:20,242 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:20,242 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:20,245 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:20,245 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:20,245 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:20,245 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:20,245 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:20,247 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,247 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:20,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,247 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:20,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,249 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:20,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:20,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:20,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:20,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:20,251 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:20,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:20,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:20,251 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:20,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,251 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:20,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,251 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:20,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,279 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/ee643df65a0046b1952dfd5d27a1d994 2023-04-03 05:56:20,293 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/963f161354934dce83697a7d2b69bf8c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/963f161354934dce83697a7d2b69bf8c 2023-04-03 05:56:20,296 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/963f161354934dce83697a7d2b69bf8c, entries=200, sequenceid=52, filesize=31.9 K 2023-04-03 05:56:20,297 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/68bd3cf85f474b6aa02e1d22475b2f89 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/68bd3cf85f474b6aa02e1d22475b2f89 2023-04-03 05:56:20,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39388 deadline: 1680501440312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39462 deadline: 1680501440318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39448 deadline: 1680501440319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,320 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/68bd3cf85f474b6aa02e1d22475b2f89, entries=150, sequenceid=52, filesize=11.6 K 2023-04-03 05:56:20,321 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/ee643df65a0046b1952dfd5d27a1d994 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ee643df65a0046b1952dfd5d27a1d994 2023-04-03 05:56:20,325 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ee643df65a0046b1952dfd5d27a1d994, entries=150, sequenceid=52, filesize=11.6 K 2023-04-03 05:56:20,325 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 67f8cb6d380f875bf3bd14331600fad6 in 545ms, sequenceid=52, compaction requested=false 2023-04-03 05:56:20,325 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:20,352 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:20,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:20,354 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:20,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:20,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:20,356 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:20,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:20,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:20,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:20,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,359 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:20,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:20,359 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:20,360 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:20,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:20,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:20,376 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:20,376 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:20,376 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:20,376 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:20,377 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:20,377 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:20,377 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:20,377 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:20,377 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:20,377 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:20,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:20,409 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:20,423 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403b981475b2afb40f98a18fa23bd6369bd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b981475b2afb40f98a18fa23bd6369bd 2023-04-03 05:56:20,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501380435, max timestamp=1680501380446Num uniques 859; 2023-04-03 05:56:20,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501380435, max timestamp=1680501380446Num uniques 859; 2023-04-03 05:56:20,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:20,449 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:20,450 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501380435, max timestamp=1680501380446Num uniques 859; 2023-04-03 05:56:20,451 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:20,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501440483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501440484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501440588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,591 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501440588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501440794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,796 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501440795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39388 deadline: 1680501440818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,823 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39448 deadline: 1680501440822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,825 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:20,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39462 deadline: 1680501440824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,837 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=62, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/be8885a52d704d8b850ec9fb092d3de6 2023-04-03 05:56:20,849 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/f10256f307874149bc27ef674ed18de3 2023-04-03 05:56:20,865 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=62 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/b7099784cce1474ba600262025c938a8 2023-04-03 05:56:20,869 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/be8885a52d704d8b850ec9fb092d3de6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/be8885a52d704d8b850ec9fb092d3de6 2023-04-03 05:56:20,873 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/be8885a52d704d8b850ec9fb092d3de6, entries=150, sequenceid=62, filesize=25.1 K 2023-04-03 05:56:20,875 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/f10256f307874149bc27ef674ed18de3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/f10256f307874149bc27ef674ed18de3 2023-04-03 05:56:20,884 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/f10256f307874149bc27ef674ed18de3, entries=150, sequenceid=62, filesize=11.6 K 2023-04-03 05:56:20,885 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/b7099784cce1474ba600262025c938a8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b7099784cce1474ba600262025c938a8 2023-04-03 05:56:20,889 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b7099784cce1474ba600262025c938a8, entries=150, sequenceid=62, filesize=11.6 K 2023-04-03 05:56:20,891 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 67f8cb6d380f875bf3bd14331600fad6 in 515ms, sequenceid=62, compaction requested=true 2023-04-03 05:56:20,891 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:20,891 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:20,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:20,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:20,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:20,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:20,893 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,893 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:20,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:20,894 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,894 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:20,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:20,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:20,896 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11fbb62f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:20,896 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:20,896 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:20,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:20,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:20,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:20,899 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:20,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,899 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:20,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:20,899 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:20,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:20,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:20,902 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:20,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:20,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:20,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:20,902 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:20,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:20,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:20,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:20,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:20,905 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,907 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:20,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:20,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:20,909 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:20,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:20,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:20,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:20,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:20,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:20,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:20,910 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:20,910 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:20,910 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:20,910 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:20,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:20,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:20,959 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-03 05:56:21,010 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:21,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:21,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:21,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:21,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:21,015 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:21,015 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,015 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:21,015 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:21,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,017 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:21,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:21,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:21,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:21,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:21,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:21,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:21,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:21,020 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:21,020 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,020 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:21,020 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-03 05:56:21,020 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:21,021 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:21,060 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304035efe2424cb954321b01e63ab884c07f9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304035efe2424cb954321b01e63ab884c07f9 2023-04-03 05:56:21,066 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/febc4f8da53b41f1aee7c5b2e949c38d 2023-04-03 05:56:21,077 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/1434b8975ce24ba8991e2e5aca1c5906 2023-04-03 05:56:21,089 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/18f516b13e854a598fb07b46eb7c7a9a 2023-04-03 05:56:21,093 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/febc4f8da53b41f1aee7c5b2e949c38d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/febc4f8da53b41f1aee7c5b2e949c38d 2023-04-03 05:56:21,096 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/febc4f8da53b41f1aee7c5b2e949c38d, entries=150, sequenceid=98, filesize=25.1 K 2023-04-03 05:56:21,096 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/1434b8975ce24ba8991e2e5aca1c5906 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1434b8975ce24ba8991e2e5aca1c5906 2023-04-03 05:56:21,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:21,101 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:21,101 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1434b8975ce24ba8991e2e5aca1c5906, entries=150, sequenceid=98, filesize=11.6 K 2023-04-03 05:56:21,102 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/18f516b13e854a598fb07b46eb7c7a9a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/18f516b13e854a598fb07b46eb7c7a9a 2023-04-03 05:56:21,106 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/18f516b13e854a598fb07b46eb7c7a9a, entries=150, sequenceid=98, filesize=11.6 K 2023-04-03 05:56:21,106 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=6.71 KB/6870 for 67f8cb6d380f875bf3bd14331600fad6 in 86ms, sequenceid=98, compaction requested=true 2023-04-03 05:56:21,107 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:21,107 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:21,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:21,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:21,107 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:21,116 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,116 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,116 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,117 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:21,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@438a2b75[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:21,118 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:21,118 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,120 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:21,120 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,121 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,122 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:21,122 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:21,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:21,124 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:21,124 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:21,124 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:21,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:21,125 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:21,126 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,126 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,126 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,127 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,128 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,131 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,131 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,131 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,131 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:21,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,131 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,131 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,131 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,131 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:21,132 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:21,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,131 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:21,148 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:21,148 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:21,148 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:21,148 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:21,148 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:21,148 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:21,148 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:21,178 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403bd3346e23ac5444e81ab1985c3d4f210 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403bd3346e23ac5444e81ab1985c3d4f210 2023-04-03 05:56:21,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501381167, max timestamp=1680501381202Num uniques 859; 2023-04-03 05:56:21,204 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:21,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501381167, max timestamp=1680501381202Num uniques 859; 2023-04-03 05:56:21,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501381167, max timestamp=1680501381202Num uniques 859; 2023-04-03 05:56:21,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:21,205 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:21,236 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:21,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:21,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:21,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:21,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:21,240 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:21,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:21,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:21,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,241 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,243 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,243 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,243 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,243 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:21,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,244 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:21,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,245 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:21,245 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:21,245 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:21,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:21,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501441248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,251 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:21,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501441248, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:21,252 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,252 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:21,252 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:21,252 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:21,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:21,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,254 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:21,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@309368be[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:21,256 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:21,256 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,257 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:21,257 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:21,258 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,258 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,260 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:21,260 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:21,263 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:21,263 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:21,263 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:21,263 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:21,263 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:21,263 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:21,269 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,269 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,269 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,270 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,273 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,273 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,273 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,273 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,273 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,273 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,274 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,274 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,274 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,273 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:21,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:21,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:21,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,274 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:21,274 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:21,354 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:21,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501441352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:21,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501441352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,374 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:21,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:21,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:21,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:21,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:21,385 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:21,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:21,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:21,386 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,386 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,387 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,387 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,387 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,388 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:21,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:21,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:21,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:21,392 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:21,392 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,392 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:21,392 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:21,392 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:21,392 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,392 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:21,393 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:21,393 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,393 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:21,393 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:21,393 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,393 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:21,393 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:21,398 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,398 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,398 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,398 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:21,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b696f16[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:21,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:21,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,404 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:21,404 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,404 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:21,404 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:21,406 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,406 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:21,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:21,410 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:21,410 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:21,410 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:21,410 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:21,410 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:21,411 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,411 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,411 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,413 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,415 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:21,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:21,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:21,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,415 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:21,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,415 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:21,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,415 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,416 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,416 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,416 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,516 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:21,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:21,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:21,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:21,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:21,520 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:21,520 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,520 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:21,520 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:21,520 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,521 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,522 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,523 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,523 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:21,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:21,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:21,524 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:21,524 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,524 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:21,528 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:21,528 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,528 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:21,528 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:21,528 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:21,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:21,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,530 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,531 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,531 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:21,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@476ffa9a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:21,533 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:21,533 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,534 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,534 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:21,534 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:21,534 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,534 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:21,534 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:21,535 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,535 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:21,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:21,538 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:21,538 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:21,538 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:21,538 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:21,538 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:21,540 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,540 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,541 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,541 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,542 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,544 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:21,544 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:21,544 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,545 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:21,545 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:21,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:21,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:21,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501441557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:21,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501441557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,609 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=108, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/0de10612b7324fd28a7edeb3f0e45317 2023-04-03 05:56:21,626 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/ff5149cd32364b2d80acd45f8ae179f6 2023-04-03 05:56:21,645 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:21,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:21,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:21,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:21,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:21,652 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:21,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:21,652 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:21,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,655 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,655 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,656 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,656 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:21,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:21,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:21,660 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:21,660 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,660 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:21,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:21,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:21,664 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,664 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:21,664 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:21,664 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:21,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:21,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:21,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:21,666 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,666 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,666 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,666 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:21,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55acb081[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:21,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:21,668 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,672 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,672 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:21,672 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:21,672 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:21,672 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,672 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:21,674 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:21,674 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,674 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,674 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:21,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:21,677 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:21,677 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:21,677 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:21,677 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:21,677 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:21,678 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,678 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,678 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,679 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,680 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,692 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,692 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,692 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:21,693 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:21,693 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:21,693 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:21,693 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:21,793 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:21,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:21,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:21,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:21,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:21,801 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:21,801 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,801 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:21,802 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:21,802 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,802 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,803 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,804 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,804 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,804 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:21,805 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:21,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:21,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,805 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:21,806 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:21,806 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:21,807 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,807 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,808 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,808 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:21,810 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e51e412[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:21,810 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:21,810 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,811 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,811 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:21,811 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:21,811 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:21,811 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,811 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:21,813 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:21,813 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,813 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:21,813 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:21,818 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:21,818 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:21,818 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:21,818 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:21,818 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:21,818 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:21,820 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,820 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,824 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,826 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:21,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39448 deadline: 1680501441825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,828 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,828 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,828 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,828 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,828 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,828 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:21,828 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,829 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,829 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:21,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:21,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,829 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,829 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:21,829 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:21,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:21,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39462 deadline: 1680501441828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,832 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:21,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39388 deadline: 1680501441829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:21,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501441862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,866 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:21,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501441863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,929 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:21,930 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:21,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:21,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:21,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:21,933 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:21,933 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,933 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:21,933 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:21,933 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,933 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,935 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,935 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,936 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,936 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,936 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,937 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:21,937 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:21,937 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:21,937 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:21,938 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,938 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:21,938 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:21,938 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:21,938 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:21,941 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,941 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,941 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,941 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:21,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e803511[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:21,943 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:21,943 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,944 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,945 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,945 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:21,945 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:21,945 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:21,945 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:21,946 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,946 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:21,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:21,949 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:21,949 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:21,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:21,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:21,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:21,949 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:21,951 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,951 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,951 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:21,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:21,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:21,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,952 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:21,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,953 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:21,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:21,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,956 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:21,956 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:21,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:21,956 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:21,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:21,956 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:21,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:21,956 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,956 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:21,956 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:21,956 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,956 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:21,956 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:21,956 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:21,956 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:21,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:22,038 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/cc934d3a535743c787ed56ae4aba47e0 2023-04-03 05:56:22,042 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/0de10612b7324fd28a7edeb3f0e45317 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/0de10612b7324fd28a7edeb3f0e45317 2023-04-03 05:56:22,046 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/0de10612b7324fd28a7edeb3f0e45317, entries=350, sequenceid=108, filesize=52.3 K 2023-04-03 05:56:22,047 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/ff5149cd32364b2d80acd45f8ae179f6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ff5149cd32364b2d80acd45f8ae179f6 2023-04-03 05:56:22,051 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ff5149cd32364b2d80acd45f8ae179f6, entries=100, sequenceid=108, filesize=9.3 K 2023-04-03 05:56:22,052 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/cc934d3a535743c787ed56ae4aba47e0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/cc934d3a535743c787ed56ae4aba47e0 2023-04-03 05:56:22,055 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/cc934d3a535743c787ed56ae4aba47e0, entries=100, sequenceid=108, filesize=9.3 K 2023-04-03 05:56:22,056 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 67f8cb6d380f875bf3bd14331600fad6 in 908ms, sequenceid=108, compaction requested=true 2023-04-03 05:56:22,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:22,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:22,056 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:56:22,056 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:56:22,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:22,056 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:22,057 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:22,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:22,058 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 163430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:56:22,058 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/A is initiating minor compaction (all files) 2023-04-03 05:56:22,058 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/A in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:22,058 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/b12c3337eeb14917a787387c4cddb0b7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/963f161354934dce83697a7d2b69bf8c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/be8885a52d704d8b850ec9fb092d3de6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/febc4f8da53b41f1aee7c5b2e949c38d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/0de10612b7324fd28a7edeb3f0e45317] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=159.6 K 2023-04-03 05:56:22,058 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:56:22,059 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/B is initiating minor compaction (all files) 2023-04-03 05:56:22,059 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting b12c3337eeb14917a787387c4cddb0b7, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680501379650 2023-04-03 05:56:22,059 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/B in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:22,059 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/a0a8f78c142146809182a67a9514b480, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/68bd3cf85f474b6aa02e1d22475b2f89, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/f10256f307874149bc27ef674ed18de3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1434b8975ce24ba8991e2e5aca1c5906, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ff5149cd32364b2d80acd45f8ae179f6] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=55.7 K 2023-04-03 05:56:22,059 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 963f161354934dce83697a7d2b69bf8c, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680501379689 2023-04-03 05:56:22,059 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a0a8f78c142146809182a67a9514b480, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680501379650 2023-04-03 05:56:22,060 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting be8885a52d704d8b850ec9fb092d3de6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680501379781 2023-04-03 05:56:22,060 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 68bd3cf85f474b6aa02e1d22475b2f89, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680501379689 2023-04-03 05:56:22,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:22,060 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting f10256f307874149bc27ef674ed18de3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680501379781 2023-04-03 05:56:22,060 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting febc4f8da53b41f1aee7c5b2e949c38d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680501380478 2023-04-03 05:56:22,061 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 0de10612b7324fd28a7edeb3f0e45317, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680501381098 2023-04-03 05:56:22,061 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1434b8975ce24ba8991e2e5aca1c5906, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680501380478 2023-04-03 05:56:22,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:22,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:22,061 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:22,061 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:22,062 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:22,062 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:22,062 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ff5149cd32364b2d80acd45f8ae179f6, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680501381143 2023-04-03 05:56:22,062 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:22,062 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:22,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:22,063 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:22,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:22,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:22,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:22,063 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:22,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:22,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:22,065 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:22,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:22,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:22,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:22,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:22,068 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:22,082 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#B#compaction#567 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:22,088 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:22,088 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:22,088 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:22,088 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:22,089 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-03 05:56:22,089 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:22,089 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:22,089 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:22,089 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:22,089 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:22,089 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:22,095 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#A#compaction#566 average throughput is 0.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:22,104 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/026a7f3aab984378951ad43674817a3a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/026a7f3aab984378951ad43674817a3a 2023-04-03 05:56:22,108 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/B of 67f8cb6d380f875bf3bd14331600fad6 into 026a7f3aab984378951ad43674817a3a(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:22,108 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:22,108 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/B, priority=11, startTime=1680501382056; duration=0sec 2023-04-03 05:56:22,108 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:22,108 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:56:22,110 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 57086 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:56:22,111 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/C is initiating minor compaction (all files) 2023-04-03 05:56:22,111 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/C in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:22,111 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e1c81ec703884187b9e36b87a64859f9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ee643df65a0046b1952dfd5d27a1d994, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b7099784cce1474ba600262025c938a8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/18f516b13e854a598fb07b46eb7c7a9a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/cc934d3a535743c787ed56ae4aba47e0] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=55.7 K 2023-04-03 05:56:22,111 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e1c81ec703884187b9e36b87a64859f9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680501379650 2023-04-03 05:56:22,112 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ee643df65a0046b1952dfd5d27a1d994, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1680501379689 2023-04-03 05:56:22,116 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b7099784cce1474ba600262025c938a8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=62, earliestPutTs=1680501379781 2023-04-03 05:56:22,116 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 18f516b13e854a598fb07b46eb7c7a9a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680501380478 2023-04-03 05:56:22,117 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting cc934d3a535743c787ed56ae4aba47e0, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680501381143 2023-04-03 05:56:22,133 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#C#compaction#569 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:22,158 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/e91a0f935d7d49079bbd5a37a680aae2 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e91a0f935d7d49079bbd5a37a680aae2 2023-04-03 05:56:22,162 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/C of 67f8cb6d380f875bf3bd14331600fad6 into e91a0f935d7d49079bbd5a37a680aae2(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:22,162 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:22,162 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/C, priority=11, startTime=1680501382056; duration=0sec 2023-04-03 05:56:22,162 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:22,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:22,369 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:22,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:22,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501442379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:22,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:22,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501442382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:22,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:22,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501442484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:22,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:22,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501442487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:22,531 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304032c8d8b5a685f457c8c6e70448f74eccf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032c8d8b5a685f457c8c6e70448f74eccf 2023-04-03 05:56:22,688 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:22,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501442687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:22,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:22,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501442690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:22,959 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=144, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/d45d11adef8849ebae46c3de1159542a 2023-04-03 05:56:22,970 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/06b86f42aec6439abe2cb93b497b4993 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/06b86f42aec6439abe2cb93b497b4993 2023-04-03 05:56:22,974 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/ae605e9fdde04b4dbdb0421768758184 2023-04-03 05:56:22,974 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/A of 67f8cb6d380f875bf3bd14331600fad6 into 06b86f42aec6439abe2cb93b497b4993(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:22,974 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:22,974 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/A, priority=11, startTime=1680501382056; duration=0sec 2023-04-03 05:56:22,974 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:22,985 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=144 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/07ed40064389470fac5bd5328b6072ad 2023-04-03 05:56:22,989 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/d45d11adef8849ebae46c3de1159542a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d45d11adef8849ebae46c3de1159542a 2023-04-03 05:56:22,992 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d45d11adef8849ebae46c3de1159542a, entries=150, sequenceid=144, filesize=25.3 K 2023-04-03 05:56:22,992 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/ae605e9fdde04b4dbdb0421768758184 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ae605e9fdde04b4dbdb0421768758184 2023-04-03 05:56:22,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:22,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501442991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:22,995 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ae605e9fdde04b4dbdb0421768758184, entries=150, sequenceid=144, filesize=11.8 K 2023-04-03 05:56:22,996 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/07ed40064389470fac5bd5328b6072ad as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/07ed40064389470fac5bd5328b6072ad 2023-04-03 05:56:22,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:22,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501442996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:22,998 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/07ed40064389470fac5bd5328b6072ad, entries=150, sequenceid=144, filesize=11.8 K 2023-04-03 05:56:22,999 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 67f8cb6d380f875bf3bd14331600fad6 in 910ms, sequenceid=144, compaction requested=false 2023-04-03 05:56:22,999 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:22,999 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:22,999 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:22,999 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:22,999 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:22,999 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:22,999 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:23,001 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:23,001 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:23,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:23,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:23,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@683a371[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:23,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:23,003 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:23,004 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:23,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:23,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:23,004 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:23,004 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:23,004 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:23,007 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:23,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:23,007 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:23,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:23,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:23,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:23,010 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:23,010 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:23,010 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:23,010 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:23,010 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:23,011 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:23,011 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:23,011 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:23,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:23,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:23,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:23,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:23,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:23,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:23,014 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:23,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:23,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:23,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:23,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:23,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:23,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:23,021 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:23,021 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:23,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:23,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:23,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:23,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:23,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:23,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:23,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:23,121 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:23,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:23,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:23,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:23,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:23,124 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:23,124 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:23,124 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:23,124 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:23,124 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:23,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:23,127 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:23,127 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:23,127 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:23,127 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:23,128 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:23,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:23,128 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:23,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:23,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:23,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:23,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:23,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:23,129 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:23,129 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:23,129 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:23,129 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:23,129 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:23,129 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:23,129 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:23,129 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:23,129 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:23,129 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:23,129 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:23,140 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403044ed791c42e4b67ab1c4371c1092615 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403044ed791c42e4b67ab1c4371c1092615 2023-04-03 05:56:23,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:23,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:23,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501383519, max timestamp=1680501383524Num uniques 859; 2023-04-03 05:56:23,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:23,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501383519, max timestamp=1680501383524Num uniques 859; 2023-04-03 05:56:23,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501383519, max timestamp=1680501383524Num uniques 859; 2023-04-03 05:56:23,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:23,526 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:23,546 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=157, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/8f21d2760dcd46c4b8cdecc93087cd99 2023-04-03 05:56:23,554 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/a9614195fbb34f589b14c4bf80a0d585 2023-04-03 05:56:23,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680501383546, max timestamp=1680501383555Num uniques 1018; 2023-04-03 05:56:23,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-03 05:56:23,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680501383546, max timestamp=1680501383555Num uniques 1018; 2023-04-03 05:56:23,557 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-03 05:56:23,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680501383546, max timestamp=1680501383555Num uniques 1018; 2023-04-03 05:56:23,558 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-03 05:56:23,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:23,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501443592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:23,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501443592, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,698 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:23,699 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501443697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,701 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:23,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501443698, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,842 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:23,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39448 deadline: 1680501443840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,844 DEBUG [Thread-4121] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4138 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:56:23,847 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:23,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39462 deadline: 1680501443845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,849 DEBUG [Thread-4113] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4139 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:56:23,863 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:23,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39388 deadline: 1680501443859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,864 DEBUG [Thread-4119] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4170 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:56:23,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:23,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39412 deadline: 1680501443901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:23,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39362 deadline: 1680501443902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:23,986 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=157 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/1bf69a3444f3458eb96e2808c0130e7c 2023-04-03 05:56:23,990 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/8f21d2760dcd46c4b8cdecc93087cd99 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8f21d2760dcd46c4b8cdecc93087cd99 2023-04-03 05:56:23,993 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8f21d2760dcd46c4b8cdecc93087cd99, entries=150, sequenceid=157, filesize=25.3 K 2023-04-03 05:56:23,994 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/a9614195fbb34f589b14c4bf80a0d585 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/a9614195fbb34f589b14c4bf80a0d585 2023-04-03 05:56:23,997 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/a9614195fbb34f589b14c4bf80a0d585, entries=150, sequenceid=157, filesize=11.8 K 2023-04-03 05:56:24,000 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/1bf69a3444f3458eb96e2808c0130e7c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/1bf69a3444f3458eb96e2808c0130e7c 2023-04-03 05:56:24,009 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/1bf69a3444f3458eb96e2808c0130e7c, entries=150, sequenceid=157, filesize=11.8 K 2023-04-03 05:56:24,010 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 67f8cb6d380f875bf3bd14331600fad6 in 880ms, sequenceid=157, compaction requested=true 2023-04-03 05:56:24,010 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:24,010 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:24,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:24,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:24,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:24,011 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,011 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,012 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:24,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@361b2398[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:24,014 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:24,014 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,015 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:24,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:24,015 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:24,015 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:24,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:24,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:24,017 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:24,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:24,021 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:24,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:24,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:24,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:24,021 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:24,024 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,025 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,025 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,025 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,026 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:24,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:24,028 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:24,029 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:24,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,029 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:24,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,130 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:24,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:24,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:24,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:24,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:24,135 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:24,135 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:24,135 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:24,135 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:24,135 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,136 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,137 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,138 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,138 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:24,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:24,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:24,140 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:24,140 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:24,140 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:24,140 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,140 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:24,140 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=134.18 KB heapSize=379.45 KB 2023-04-03 05:56:24,141 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:24,141 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:24,141 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:24,141 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:24,141 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:24,141 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:24,167 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403d0ddcb805463432b8826bf45d12d8b66 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d0ddcb805463432b8826bf45d12d8b66 2023-04-03 05:56:24,177 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=210, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/c72f11e2d2f143749dd1c96699734af9 2023-04-03 05:56:24,194 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/2e4f438169c2461b8326222fa800593c 2023-04-03 05:56:24,204 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=210 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/5d3975e77c0647f0adfd7c2f65fbc930 2023-04-03 05:56:24,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:24,206 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:24,207 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/c72f11e2d2f143749dd1c96699734af9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/c72f11e2d2f143749dd1c96699734af9 2023-04-03 05:56:24,210 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/c72f11e2d2f143749dd1c96699734af9, entries=150, sequenceid=210, filesize=25.3 K 2023-04-03 05:56:24,211 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/2e4f438169c2461b8326222fa800593c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2e4f438169c2461b8326222fa800593c 2023-04-03 05:56:24,214 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2e4f438169c2461b8326222fa800593c, entries=150, sequenceid=210, filesize=11.8 K 2023-04-03 05:56:24,214 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/5d3975e77c0647f0adfd7c2f65fbc930 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/5d3975e77c0647f0adfd7c2f65fbc930 2023-04-03 05:56:24,217 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/5d3975e77c0647f0adfd7c2f65fbc930, entries=150, sequenceid=210, filesize=11.8 K 2023-04-03 05:56:24,218 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~379.41 KB/388512, currentSize=33.54 KB/34350 for 67f8cb6d380f875bf3bd14331600fad6 in 78ms, sequenceid=210, compaction requested=true 2023-04-03 05:56:24,218 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:24,218 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:24,218 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:24,219 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,219 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,220 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,220 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:24,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:24,222 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:24,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:24,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:24,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:24,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:24,222 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:24,222 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:24,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@471b203d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:24,224 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:24,224 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,230 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:24,231 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:24,231 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,231 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:24,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:24,232 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,232 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:24,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:24,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:24,235 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:24,235 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:24,235 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:24,235 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:24,235 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:24,236 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,236 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,236 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,237 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,238 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:24,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:24,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,240 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:24,240 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:24,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,240 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501384249, max timestamp=1680501384257Num uniques 859; 2023-04-03 05:56:24,258 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:24,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501384249, max timestamp=1680501384257Num uniques 859; 2023-04-03 05:56:24,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501384249, max timestamp=1680501384257Num uniques 859; 2023-04-03 05:56:24,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:24,259 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:24,289 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:24,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:24,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501444286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501444285, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,340 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:24,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:24,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:24,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:24,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:24,344 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:24,344 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:24,344 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:24,344 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:24,344 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,344 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,346 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:24,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:24,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:24,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:24,349 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,349 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:24,349 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:24,349 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:24,349 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,350 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:24,350 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,351 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:24,353 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,353 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,353 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:24,353 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d88d5a5[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:24,353 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:24,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:24,354 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:24,355 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:24,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:24,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:24,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:24,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:24,361 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:24,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:24,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:24,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:24,361 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:24,368 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,369 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,369 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,370 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,375 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:24,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:24,376 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:24,376 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:24,375 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:24,376 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:24,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501444390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,393 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:24,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501444391, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,476 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:24,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:24,479 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:24,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:24,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:24,480 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:24,480 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:24,480 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:24,480 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:24,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,482 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,483 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:24,483 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:24,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,484 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:24,484 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:24,484 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:24,484 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:24,484 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,484 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:24,485 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:24,485 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:24,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:24,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:24,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:24,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,490 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,490 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,490 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:24,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@780cbd72[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:24,492 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:24,492 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,493 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,493 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:24,493 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:24,493 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:24,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,494 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:24,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:24,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,495 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,495 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:24,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:24,499 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:24,499 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:24,499 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:24,499 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:24,499 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:24,501 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,501 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,501 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,513 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,515 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,520 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,520 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,520 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:24,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:24,520 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,520 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,520 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,520 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,520 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,520 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,520 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,528 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:24,528 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:24,597 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:24,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501444595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:24,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501444596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,628 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:24,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:24,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:24,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:24,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:24,632 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:24,632 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:24,632 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:24,632 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:24,632 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,632 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,634 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,634 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,634 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,634 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:24,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:24,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:24,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:24,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:24,635 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:24,635 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:24,635 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:24,635 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:24,635 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:24,635 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,635 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:24,635 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:24,636 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:24,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:24,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:24,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:24,637 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,638 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304037e14ce6955434bbcb9d1413f421ee724 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037e14ce6955434bbcb9d1413f421ee724 2023-04-03 05:56:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:24,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@630f0129[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:24,639 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:24,639 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,640 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:24,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:24,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:24,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,642 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:24,643 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,643 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:24,643 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:24,643 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:24,643 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:24,643 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:24,643 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:24,643 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:24,643 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:24,644 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,645 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,645 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,645 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,648 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,648 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:24,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:24,648 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:24,648 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,648 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,648 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,648 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:24,648 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,648 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:24,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,648 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,649 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,649 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,748 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:24,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:24,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:24,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:24,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:24,752 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:24,752 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:24,752 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:24,752 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:24,752 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,752 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,753 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,753 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,753 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,753 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:24,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:24,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:24,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:24,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:24,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:24,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:24,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:24,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:24,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:24,755 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,755 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:24,755 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:24,755 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:24,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:24,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:24,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:24,757 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,757 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,757 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,757 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:24,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:24,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ce7f933[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:24,759 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:24,759 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,760 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,760 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:24,760 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:24,760 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:24,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:24,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,761 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:24,761 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,761 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:24,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:24,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:24,763 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:24,763 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:24,763 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:24,763 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:24,764 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:24,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,766 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,767 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,771 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:24,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:24,771 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,771 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:24,772 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:24,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,872 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:24,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:24,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:24,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:24,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:24,876 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:24,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:24,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:24,876 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:24,877 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,877 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,878 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,878 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,878 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,878 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:24,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:24,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:24,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:24,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:24,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:24,884 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,884 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:24,884 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:24,884 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:24,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:24,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:24,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:24,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:24,886 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,886 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,891 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:24,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:24,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12632a5a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:24,893 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:24,893 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:24,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:24,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:24,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:24,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,899 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:24,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,899 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,899 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:24,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:24,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:24,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501444900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:24,903 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:24,903 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:24,903 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:24,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:24,903 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:24,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501444901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,903 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:24,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,905 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:24,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,905 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:24,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:24,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:24,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,906 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:24,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:24,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:24,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:24,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:24,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:24,909 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:24,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:24,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:24,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:24,909 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:24,909 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:24,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:24,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:24,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,009 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:25,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:25,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:25,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:25,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:25,013 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:25,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:25,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:25,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:25,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,015 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,016 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:25,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:25,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:25,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,017 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:25,017 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:25,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:25,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:25,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:25,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:25,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:25,017 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:25,017 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:25,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:25,017 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:25,017 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:25,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:25,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:25,017 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:25,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:25,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,019 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:25,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:25,019 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,019 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:25,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:25,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:25,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:25,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:25,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f99f249[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:25,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:25,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:25,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:25,022 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,022 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:25,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:25,023 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:25,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:25,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:25,023 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:25,025 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:25,025 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:25,026 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:25,026 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:25,026 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:25,026 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:25,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:25,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,028 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:25,028 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:25,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,030 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,032 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,032 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:25,032 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,032 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:25,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:25,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:25,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:25,032 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:25,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:25,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:25,032 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:25,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:25,032 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,032 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:25,032 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:25,032 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,033 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,033 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,043 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=220, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/e5084b3efc90467398833aa9dfeee814 2023-04-03 05:56:25,051 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/037ef8740b194b149590b56281b717f7 2023-04-03 05:56:25,133 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:25,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:25,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:25,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:25,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:25,135 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:25,135 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:25,135 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:25,135 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:25,136 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,136 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,137 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:25,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:25,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:25,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:25,138 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:25,138 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:25,138 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:25,138 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:25,138 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:25,138 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:25,138 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:25,138 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:25,138 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:25,138 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:25,138 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:25,139 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:25,139 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:25,139 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,139 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:25,139 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:25,140 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,140 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:25,140 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,140 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:25,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:25,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:25,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1404acda[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:25,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:25,142 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,144 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,144 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:25,144 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:25,144 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:25,144 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,144 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:25,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:25,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:25,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:25,145 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:25,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:25,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:25,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:25,147 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:25,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:25,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:25,147 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:25,147 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:25,149 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,149 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,149 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:25,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,149 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,150 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:25,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:25,153 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:25,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:25,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:25,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:25,153 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:25,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:25,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:25,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,153 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:25,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,153 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:25,253 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:25,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:25,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:25,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:25,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:25,257 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:25,257 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:25,257 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:25,257 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:25,257 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,257 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,259 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:25,259 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:25,260 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,260 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:25,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:25,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:25,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:25,261 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:25,261 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:25,261 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:25,261 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:25,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:25,262 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:25,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,263 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:25,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:25,264 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,264 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:25,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:25,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2719e108[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:25,265 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:25,265 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,266 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,267 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:25,267 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:25,267 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:25,267 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,267 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:25,271 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:25,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:25,274 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:25,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:25,274 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:25,274 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:25,275 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:25,275 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:25,275 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:25,275 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:25,276 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,276 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:25,276 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:25,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,277 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,279 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:25,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:25,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:25,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:25,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:25,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:25,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:25,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:25,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,282 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,282 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,282 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,282 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,282 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:25,282 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:25,382 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:25,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:25,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:25,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:25,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:25,385 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:25,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:25,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:25,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:25,386 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,386 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,387 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:25,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:25,387 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,388 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,388 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:25,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:25,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:25,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:25,389 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:25,389 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:25,389 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:25,389 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:25,389 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:25,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:25,392 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,392 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:25,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:25,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:25,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:25,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ca32e4b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:25,395 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:25,395 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,397 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,397 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,397 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:25,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:25,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:25,397 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:25,398 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:25,398 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:25,398 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,399 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:25,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:25,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:25,403 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:25,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:25,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:25,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:25,403 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:25,405 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,405 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:25,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,405 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:25,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:25,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,406 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:25,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,408 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:25,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501445406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:25,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501445406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,414 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,414 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,414 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,414 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,414 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,414 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:25,414 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,414 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:25,414 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:25,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:25,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:25,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:25,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:25,414 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:25,415 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:25,415 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:25,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:25,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:25,477 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=220 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/407e05ccd0e045858c2253fa2b826d00 2023-04-03 05:56:25,481 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/e5084b3efc90467398833aa9dfeee814 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e5084b3efc90467398833aa9dfeee814 2023-04-03 05:56:25,484 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e5084b3efc90467398833aa9dfeee814, entries=350, sequenceid=220, filesize=52.6 K 2023-04-03 05:56:25,484 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/037ef8740b194b149590b56281b717f7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/037ef8740b194b149590b56281b717f7 2023-04-03 05:56:25,487 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/037ef8740b194b149590b56281b717f7, entries=150, sequenceid=220, filesize=11.8 K 2023-04-03 05:56:25,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/407e05ccd0e045858c2253fa2b826d00 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/407e05ccd0e045858c2253fa2b826d00 2023-04-03 05:56:25,491 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/407e05ccd0e045858c2253fa2b826d00, entries=150, sequenceid=220, filesize=11.8 K 2023-04-03 05:56:25,491 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 67f8cb6d380f875bf3bd14331600fad6 in 1269ms, sequenceid=220, compaction requested=true 2023-04-03 05:56:25,491 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:25,492 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:25,492 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:56:25,492 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:56:25,492 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:25,492 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:25,493 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 157466 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:56:25,493 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60202 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:56:25,493 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/B is initiating minor compaction (all files) 2023-04-03 05:56:25,493 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/B in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:25,493 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/026a7f3aab984378951ad43674817a3a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ae605e9fdde04b4dbdb0421768758184, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/a9614195fbb34f589b14c4bf80a0d585, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2e4f438169c2461b8326222fa800593c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/037ef8740b194b149590b56281b717f7] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=58.8 K 2023-04-03 05:56:25,493 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/A is initiating minor compaction (all files) 2023-04-03 05:56:25,494 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/A in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:25,494 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/06b86f42aec6439abe2cb93b497b4993, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d45d11adef8849ebae46c3de1159542a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8f21d2760dcd46c4b8cdecc93087cd99, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/c72f11e2d2f143749dd1c96699734af9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e5084b3efc90467398833aa9dfeee814] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=153.8 K 2023-04-03 05:56:25,494 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 026a7f3aab984378951ad43674817a3a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680501380478 2023-04-03 05:56:25,494 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 06b86f42aec6439abe2cb93b497b4993, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680501380478 2023-04-03 05:56:25,494 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ae605e9fdde04b4dbdb0421768758184, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680501381236 2023-04-03 05:56:25,494 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting d45d11adef8849ebae46c3de1159542a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680501381236 2023-04-03 05:56:25,495 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a9614195fbb34f589b14c4bf80a0d585, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680501382377 2023-04-03 05:56:25,495 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8f21d2760dcd46c4b8cdecc93087cd99, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680501382377 2023-04-03 05:56:25,495 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 2e4f438169c2461b8326222fa800593c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680501383578 2023-04-03 05:56:25,495 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting c72f11e2d2f143749dd1c96699734af9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680501383578 2023-04-03 05:56:25,495 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 037ef8740b194b149590b56281b717f7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680501384215 2023-04-03 05:56:25,495 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e5084b3efc90467398833aa9dfeee814, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680501384205 2023-04-03 05:56:25,503 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#B#compaction#582 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:25,506 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#A#compaction#581 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:25,515 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:25,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:25,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:25,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:25,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:25,524 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:25,524 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:25,524 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:25,524 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:25,525 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,525 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,526 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:25,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:25,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,528 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:25,528 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:25,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:25,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:25,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:25,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:25,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:25,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:25,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:25,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:25,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:25,536 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/777386cc7e6345c1a70f28c369b6560b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/777386cc7e6345c1a70f28c369b6560b 2023-04-03 05:56:25,539 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:25,539 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:25,540 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-03 05:56:25,540 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:25,540 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/B of 67f8cb6d380f875bf3bd14331600fad6 into 777386cc7e6345c1a70f28c369b6560b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:25,540 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:25,540 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:25,540 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:25,540 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/B, priority=11, startTime=1680501385492; duration=0sec 2023-04-03 05:56:25,540 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:25,540 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:25,540 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:25,540 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:25,541 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:56:25,542 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60202 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:56:25,542 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/C is initiating minor compaction (all files) 2023-04-03 05:56:25,542 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/C in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:25,542 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e91a0f935d7d49079bbd5a37a680aae2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/07ed40064389470fac5bd5328b6072ad, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/1bf69a3444f3458eb96e2808c0130e7c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/5d3975e77c0647f0adfd7c2f65fbc930, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/407e05ccd0e045858c2253fa2b826d00] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=58.8 K 2023-04-03 05:56:25,542 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e91a0f935d7d49079bbd5a37a680aae2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1680501380478 2023-04-03 05:56:25,543 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 07ed40064389470fac5bd5328b6072ad, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=144, earliestPutTs=1680501381236 2023-04-03 05:56:25,543 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1bf69a3444f3458eb96e2808c0130e7c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=157, earliestPutTs=1680501382377 2023-04-03 05:56:25,543 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 5d3975e77c0647f0adfd7c2f65fbc930, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=210, earliestPutTs=1680501383578 2023-04-03 05:56:25,544 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 407e05ccd0e045858c2253fa2b826d00, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680501384215 2023-04-03 05:56:25,554 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#C#compaction#584 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:25,577 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/8e6fdce74d1e472ca2f41995f75172eb as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8e6fdce74d1e472ca2f41995f75172eb 2023-04-03 05:56:25,582 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/A of 67f8cb6d380f875bf3bd14331600fad6 into 8e6fdce74d1e472ca2f41995f75172eb(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:25,582 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:25,582 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/A, priority=11, startTime=1680501385492; duration=0sec 2023-04-03 05:56:25,582 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:25,594 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/e01de912468e4267929d78808739b733 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e01de912468e4267929d78808739b733 2023-04-03 05:56:25,599 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/C of 67f8cb6d380f875bf3bd14331600fad6 into e01de912468e4267929d78808739b733(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:25,599 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:25,599 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/C, priority=11, startTime=1680501385492; duration=0sec 2023-04-03 05:56:25,599 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:25,988 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040321959185de94484fbb57ba90a0fd164f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040321959185de94484fbb57ba90a0fd164f 2023-04-03 05:56:26,393 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=257, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/02140f2988734197a3d5cc82d0132d88 2023-04-03 05:56:26,404 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/1fb2165687f347768337acf628462a92 2023-04-03 05:56:26,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:26,412 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:26,425 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:26,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501446423, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,426 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=257 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/119a0ad1b9384cc99e1c5251944a1c81 2023-04-03 05:56:26,428 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:26,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501446426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,429 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/02140f2988734197a3d5cc82d0132d88 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/02140f2988734197a3d5cc82d0132d88 2023-04-03 05:56:26,434 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/02140f2988734197a3d5cc82d0132d88, entries=150, sequenceid=257, filesize=25.3 K 2023-04-03 05:56:26,434 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/1fb2165687f347768337acf628462a92 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1fb2165687f347768337acf628462a92 2023-04-03 05:56:26,437 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1fb2165687f347768337acf628462a92, entries=150, sequenceid=257, filesize=11.8 K 2023-04-03 05:56:26,438 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/119a0ad1b9384cc99e1c5251944a1c81 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/119a0ad1b9384cc99e1c5251944a1c81 2023-04-03 05:56:26,441 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/119a0ad1b9384cc99e1c5251944a1c81, entries=150, sequenceid=257, filesize=11.8 K 2023-04-03 05:56:26,442 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 67f8cb6d380f875bf3bd14331600fad6 in 903ms, sequenceid=257, compaction requested=false 2023-04-03 05:56:26,442 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:26,442 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:26,442 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:26,442 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:26,442 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,442 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:26,442 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:26,444 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,444 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:26,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ce88a2c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:26,447 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:26,447 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,448 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,449 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:26,449 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:26,449 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:26,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,454 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:26,454 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:26,454 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:26,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:26,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:26,454 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:26,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:26,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:26,454 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:26,454 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:26,456 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,456 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,456 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:26,456 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,458 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:26,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:26,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:26,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:26,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:26,461 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:26,461 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:26,461 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:26,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:26,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:26,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:26,461 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:26,528 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-03 05:56:26,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:26,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:26,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:26,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:26,529 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:26,529 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:26,546 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304035a9d7551cbb94a168529c4f549a13ae2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304035a9d7551cbb94a168529c4f549a13ae2 2023-04-03 05:56:26,556 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=270, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/2b809f486b9b4529b56c1b29ebd0ccad 2023-04-03 05:56:26,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501386545, max timestamp=1680501386560Num uniques 859; 2023-04-03 05:56:26,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:26,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501386545, max timestamp=1680501386560Num uniques 859; 2023-04-03 05:56:26,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:26,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501386545, max timestamp=1680501386560Num uniques 859; 2023-04-03 05:56:26,562 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:26,562 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:26,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:26,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:26,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:26,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:26,570 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:26,570 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:26,570 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:26,570 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:26,570 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,570 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/8a07e24804954825a1a766d9806f58a9 2023-04-03 05:56:26,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:26,572 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:26,572 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,572 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:26,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,573 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:26,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:26,573 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:26,573 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:26,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:26,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:26,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:26,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:26,573 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:26,573 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:26,573 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:26,574 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:26,574 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:26,574 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:26,574 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:26,574 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,574 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:26,574 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:26,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,576 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:26,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:26,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbb8917[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:26,578 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:26,578 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,579 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,579 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:26,580 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:26,580 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:26,580 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,580 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:26,581 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:26,581 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,581 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,581 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:26,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:26,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:26,583 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:26,583 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:26,583 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:26,583 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:26,583 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:26,585 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,585 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,585 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:26,586 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,587 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,593 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:26,593 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:26,593 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,593 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,593 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:26,593 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:26,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:26,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:26,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:26,593 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,594 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,594 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:26,594 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:26,594 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:26,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:26,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:26,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501446607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,614 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:26,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501446611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,694 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:26,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:26,696 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:26,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:26,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:26,697 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:26,697 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:26,697 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:26,697 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:26,697 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,702 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,702 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:26,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:26,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:26,703 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,703 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:26,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:26,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:26,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:26,704 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:26,704 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:26,704 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:26,704 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:26,704 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:26,706 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,706 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,706 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,706 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:26,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35422209[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:26,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:26,707 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:26,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:26,709 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:26,709 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,709 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:26,711 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:26,711 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:26,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:26,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501446712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:26,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:26,713 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:26,713 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:26,713 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:26,714 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:26,714 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:26,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:26,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:26,716 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:26,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501446715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,717 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:26,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:26,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:26,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:26,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:26,719 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:26,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:26,719 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:26,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:26,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:26,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,719 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:26,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,719 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,720 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,820 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:26,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:26,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:26,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:26,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:26,823 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:26,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:26,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:26,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:26,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,823 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,825 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:26,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:26,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,825 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:26,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:26,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:26,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:26,826 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:26,827 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:26,827 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:26,827 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:26,827 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:26,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:26,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:26,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:26,827 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:26,830 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,830 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,830 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,830 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:26,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:26,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26839c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:26,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:26,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,833 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,833 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:26,833 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:26,833 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:26,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:26,834 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:26,834 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:26,834 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,834 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:26,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:26,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:26,837 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:26,837 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:26,837 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:26,837 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:26,837 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:26,838 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,838 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,838 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,839 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:26,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,840 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:26,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:26,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:26,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:26,845 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:26,846 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:26,846 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:26,846 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:26,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:26,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:26,845 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,846 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:26,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:26,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:26,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501446915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:26,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501446918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,946 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:26,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:26,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:26,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:26,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:26,949 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:26,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:26,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:26,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:26,949 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,950 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,951 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:26,951 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:26,951 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,951 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:26,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,952 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:26,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:26,952 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:26,952 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:26,952 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:26,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:26,953 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:26,953 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:26,954 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,954 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,955 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:26,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2837c05e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:26,956 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:26,956 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,957 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,957 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:26,958 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:26,958 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:26,958 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,958 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:26,960 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,960 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:26,960 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,960 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:26,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:26,963 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:26,963 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:26,963 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:26,963 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:26,963 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:26,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,965 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:26,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:26,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:26,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,966 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:26,967 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:26,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:26,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:26,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:26,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:26,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:26,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:26,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:26,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:26,969 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:26,969 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:26,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:26,970 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:26,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:26,970 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:26,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:26,970 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:26,980 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=270 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/c31181aac6004c7a8242e794bfda60b1 2023-04-03 05:56:26,983 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/2b809f486b9b4529b56c1b29ebd0ccad as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/2b809f486b9b4529b56c1b29ebd0ccad 2023-04-03 05:56:26,986 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/2b809f486b9b4529b56c1b29ebd0ccad, entries=150, sequenceid=270, filesize=25.4 K 2023-04-03 05:56:26,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/8a07e24804954825a1a766d9806f58a9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8a07e24804954825a1a766d9806f58a9 2023-04-03 05:56:26,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8a07e24804954825a1a766d9806f58a9, entries=150, sequenceid=270, filesize=11.9 K 2023-04-03 05:56:26,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/c31181aac6004c7a8242e794bfda60b1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c31181aac6004c7a8242e794bfda60b1 2023-04-03 05:56:26,994 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c31181aac6004c7a8242e794bfda60b1, entries=150, sequenceid=270, filesize=11.9 K 2023-04-03 05:56:26,994 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 67f8cb6d380f875bf3bd14331600fad6 in 466ms, sequenceid=270, compaction requested=true 2023-04-03 05:56:26,994 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:26,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:26,994 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:26,994 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:26,995 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:26,995 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:26,995 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78180 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:26,995 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/A is initiating minor compaction (all files) 2023-04-03 05:56:26,995 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36600 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:26,996 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/A in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:26,996 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/B is initiating minor compaction (all files) 2023-04-03 05:56:26,996 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8e6fdce74d1e472ca2f41995f75172eb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/02140f2988734197a3d5cc82d0132d88, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/2b809f486b9b4529b56c1b29ebd0ccad] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=76.3 K 2023-04-03 05:56:26,996 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/B in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:26,996 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/777386cc7e6345c1a70f28c369b6560b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1fb2165687f347768337acf628462a92, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8a07e24804954825a1a766d9806f58a9] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=35.7 K 2023-04-03 05:56:26,996 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8e6fdce74d1e472ca2f41995f75172eb, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680501384215 2023-04-03 05:56:26,996 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 777386cc7e6345c1a70f28c369b6560b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680501384215 2023-04-03 05:56:26,997 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 02140f2988734197a3d5cc82d0132d88, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680501384275 2023-04-03 05:56:26,997 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1fb2165687f347768337acf628462a92, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680501384275 2023-04-03 05:56:26,997 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 2b809f486b9b4529b56c1b29ebd0ccad, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680501386418 2023-04-03 05:56:26,997 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 8a07e24804954825a1a766d9806f58a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680501386418 2023-04-03 05:56:27,002 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#B#compaction#590 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:27,008 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#A#compaction#591 average throughput is 3.94 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:27,070 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:27,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:27,073 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:27,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:27,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:27,073 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:27,074 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:27,074 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:27,074 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:27,074 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,074 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,077 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:27,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:27,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:27,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:27,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:27,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:27,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:27,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:27,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:27,078 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:27,078 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:27,078 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:27,079 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-03 05:56:27,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:27,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:27,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:27,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:27,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:27,079 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:27,090 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304037b95e323883e42258cc44d967389fa7b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037b95e323883e42258cc44d967389fa7b 2023-04-03 05:56:27,095 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=305, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/68a3b73ba43e4b69b1edb27f7f3a1a4c 2023-04-03 05:56:27,105 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/2539dbf426a745acb84a9d132559bdee 2023-04-03 05:56:27,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:27,223 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:27,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501447233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501447234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,339 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501447337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,340 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501447337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,411 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/9d1f40326c2d4eefb71cfc160bed572c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9d1f40326c2d4eefb71cfc160bed572c 2023-04-03 05:56:27,415 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/B of 67f8cb6d380f875bf3bd14331600fad6 into 9d1f40326c2d4eefb71cfc160bed572c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:27,415 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:27,415 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/B, priority=13, startTime=1680501386994; duration=0sec 2023-04-03 05:56:27,415 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:27,415 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:27,416 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36600 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:27,416 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/C is initiating minor compaction (all files) 2023-04-03 05:56:27,416 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/C in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:27,416 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e01de912468e4267929d78808739b733, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/119a0ad1b9384cc99e1c5251944a1c81, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c31181aac6004c7a8242e794bfda60b1] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=35.7 K 2023-04-03 05:56:27,417 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e01de912468e4267929d78808739b733, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=220, earliestPutTs=1680501384215 2023-04-03 05:56:27,417 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 119a0ad1b9384cc99e1c5251944a1c81, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=257, earliestPutTs=1680501384275 2023-04-03 05:56:27,417 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting c31181aac6004c7a8242e794bfda60b1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680501386418 2023-04-03 05:56:27,422 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#C#compaction#595 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:27,429 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/ac7f6381d2df42d8b113461c8b977958 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ac7f6381d2df42d8b113461c8b977958 2023-04-03 05:56:27,432 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/C of 67f8cb6d380f875bf3bd14331600fad6 into ac7f6381d2df42d8b113461c8b977958(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:27,432 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:27,432 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/C, priority=13, startTime=1680501386995; duration=0sec 2023-04-03 05:56:27,433 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:27,514 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=305 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/177a09b04bb24efe830292c62fa2c289 2023-04-03 05:56:27,518 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/68a3b73ba43e4b69b1edb27f7f3a1a4c as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/68a3b73ba43e4b69b1edb27f7f3a1a4c 2023-04-03 05:56:27,521 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/68a3b73ba43e4b69b1edb27f7f3a1a4c, entries=150, sequenceid=305, filesize=25.4 K 2023-04-03 05:56:27,521 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/2539dbf426a745acb84a9d132559bdee as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2539dbf426a745acb84a9d132559bdee 2023-04-03 05:56:27,524 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2539dbf426a745acb84a9d132559bdee, entries=150, sequenceid=305, filesize=11.9 K 2023-04-03 05:56:27,525 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/177a09b04bb24efe830292c62fa2c289 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/177a09b04bb24efe830292c62fa2c289 2023-04-03 05:56:27,528 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/177a09b04bb24efe830292c62fa2c289, entries=150, sequenceid=305, filesize=11.9 K 2023-04-03 05:56:27,529 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 67f8cb6d380f875bf3bd14331600fad6 in 450ms, sequenceid=305, compaction requested=false 2023-04-03 05:56:27,529 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:27,529 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:27,536 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:27,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,536 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:27,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:27,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29c5077e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:27,538 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:27,538 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,540 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:27,540 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:27,540 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,540 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:27,542 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:27,542 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:27,543 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:27,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:27,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,545 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 05:56:27,545 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:27,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:27,545 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:27,545 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:27,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:27,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:27,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:27,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:27,546 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:27,547 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:27,547 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:27,547 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:27,547 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:27,551 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,551 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:27,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,551 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:27,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:27,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:27,551 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,553 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,556 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,556 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,556 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,556 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:27,556 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:27,556 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:27,556 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:27,556 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:27,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:27,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:27,556 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,557 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,557 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,557 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:27,557 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:27,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:27,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:27,559 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403ee6ed93b83c4401f8df6ebfba56fadbf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ee6ed93b83c4401f8df6ebfba56fadbf 2023-04-03 05:56:27,564 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=320, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/d922b092f27249caada936560e5c8738 2023-04-03 05:56:27,571 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/b643e5f1bd9c4d5ea02de2ded030a780 2023-04-03 05:56:27,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501447573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,577 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501447574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,657 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:27,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:27,659 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:27,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:27,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:27,660 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:27,660 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:27,660 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:27,660 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:27,661 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,661 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,662 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,662 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:27,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:27,662 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,662 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,663 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:27,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:27,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:27,663 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:27,663 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:27,663 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:27,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:27,664 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:27,664 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:27,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:27,664 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:27,664 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:27,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:27,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:27,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:27,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:27,665 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,665 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:27,666 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,666 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:27,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:27,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@db33a25[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:27,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:27,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,669 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:27,670 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:27,670 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,670 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:27,672 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,672 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:27,672 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:27,672 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:27,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:27,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:27,674 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:27,674 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:27,674 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:27,674 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:27,674 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:27,675 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,675 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:27,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,676 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:27,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,677 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501447678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,679 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501447678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:27,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:27,680 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:27,680 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:27,680 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:27,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:27,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:27,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:27,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:27,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:27,680 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,681 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:27,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:27,780 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:27,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:27,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:27,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:27,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:27,783 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:27,783 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:27,783 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:27,783 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:27,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,786 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,786 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:27,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:27,786 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,786 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:27,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:27,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:27,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:27,787 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:27,787 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:27,787 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:27,788 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:27,788 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:27,788 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:27,788 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:27,788 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:27,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,790 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:27,790 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,790 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:27,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:27,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@393d626c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:27,791 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:27,791 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,793 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,793 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:27,793 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:27,793 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:27,793 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:27,794 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,794 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:27,794 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:27,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:27,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:27,794 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:27,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:27,797 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:27,797 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:27,797 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:27,797 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:27,797 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:27,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:27,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:27,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:27,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,806 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:27,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:27,807 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,808 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,811 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:27,812 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:27,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,812 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:27,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:27,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:27,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:27,812 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:27,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:27,812 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:27,812 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:27,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:27,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:27,835 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/e1295a826a5a47f5adcd290b23c73603 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e1295a826a5a47f5adcd290b23c73603 2023-04-03 05:56:27,838 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/A of 67f8cb6d380f875bf3bd14331600fad6 into e1295a826a5a47f5adcd290b23c73603(size=25.9 K), total size for store is 51.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:27,838 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:27,838 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/A, priority=13, startTime=1680501386994; duration=0sec 2023-04-03 05:56:27,838 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:27,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39448 deadline: 1680501447868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,871 DEBUG [Thread-4121] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8165 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:56:27,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39388 deadline: 1680501447872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,875 DEBUG [Thread-4119] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8181 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:56:27,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501447880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501447880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,890 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:27,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39462 deadline: 1680501447888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,891 DEBUG [Thread-4113] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8182 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:56:27,913 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:27,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:27,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:27,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:27,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:27,915 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:27,916 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:27,916 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:27,916 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:27,916 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,916 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,920 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:27,920 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:27,920 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,920 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:27,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,920 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:27,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:27,921 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:27,921 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:27,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:27,921 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:27,921 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:27,922 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:27,922 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:27,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:27,924 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,925 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:27,925 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,925 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:27,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:27,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:27,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f9e7025[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:27,926 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:27,926 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,927 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:27,927 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:27,928 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:27,928 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,928 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:27,929 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:27,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:27,929 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:27,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:27,931 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:27,931 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:27,931 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:27,931 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:27,931 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:27,931 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:27,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,933 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:27,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:27,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:27,933 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:27,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,934 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:27,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:27,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,938 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:27,938 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:27,938 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:27,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:27,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:27,938 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,938 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:27,938 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:27,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:27,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:27,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:27,938 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:27,938 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,938 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:27,938 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:27,939 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:27,939 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:27,978 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/c7c239a0a07b43468bef4e599d988f37 2023-04-03 05:56:27,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/d922b092f27249caada936560e5c8738 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d922b092f27249caada936560e5c8738 2023-04-03 05:56:27,985 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d922b092f27249caada936560e5c8738, entries=150, sequenceid=320, filesize=25.4 K 2023-04-03 05:56:27,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/b643e5f1bd9c4d5ea02de2ded030a780 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b643e5f1bd9c4d5ea02de2ded030a780 2023-04-03 05:56:27,988 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b643e5f1bd9c4d5ea02de2ded030a780, entries=150, sequenceid=320, filesize=11.9 K 2023-04-03 05:56:27,989 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/c7c239a0a07b43468bef4e599d988f37 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c7c239a0a07b43468bef4e599d988f37 2023-04-03 05:56:27,992 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c7c239a0a07b43468bef4e599d988f37, entries=150, sequenceid=320, filesize=11.9 K 2023-04-03 05:56:27,992 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 67f8cb6d380f875bf3bd14331600fad6 in 447ms, sequenceid=320, compaction requested=true 2023-04-03 05:56:27,992 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:27,992 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:27,992 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:27,992 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:27,992 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:27,992 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:27,993 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78582 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:27,993 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:27,993 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/B is initiating minor compaction (all files) 2023-04-03 05:56:27,994 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/B in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:27,994 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9d1f40326c2d4eefb71cfc160bed572c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2539dbf426a745acb84a9d132559bdee, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b643e5f1bd9c4d5ea02de2ded030a780] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=36.1 K 2023-04-03 05:56:27,993 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/A is initiating minor compaction (all files) 2023-04-03 05:56:27,994 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/A in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:27,994 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e1295a826a5a47f5adcd290b23c73603, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/68a3b73ba43e4b69b1edb27f7f3a1a4c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d922b092f27249caada936560e5c8738] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=76.7 K 2023-04-03 05:56:27,994 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 9d1f40326c2d4eefb71cfc160bed572c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680501386418 2023-04-03 05:56:27,994 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e1295a826a5a47f5adcd290b23c73603, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680501386418 2023-04-03 05:56:27,994 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 2539dbf426a745acb84a9d132559bdee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680501386593 2023-04-03 05:56:27,995 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 68a3b73ba43e4b69b1edb27f7f3a1a4c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680501386593 2023-04-03 05:56:27,995 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b643e5f1bd9c4d5ea02de2ded030a780, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680501387229 2023-04-03 05:56:27,995 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting d922b092f27249caada936560e5c8738, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680501387229 2023-04-03 05:56:28,000 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#B#compaction#599 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:28,002 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#A#compaction#600 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:28,015 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/1fa961ee8be844078903a81a9cd40ae1 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/1fa961ee8be844078903a81a9cd40ae1 2023-04-03 05:56:28,019 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/A of 67f8cb6d380f875bf3bd14331600fad6 into 1fa961ee8be844078903a81a9cd40ae1(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:28,019 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:28,019 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/A, priority=13, startTime=1680501387992; duration=0sec 2023-04-03 05:56:28,019 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:28,019 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:28,020 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:28,020 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/C is initiating minor compaction (all files) 2023-04-03 05:56:28,020 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/C in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,020 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ac7f6381d2df42d8b113461c8b977958, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/177a09b04bb24efe830292c62fa2c289, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c7c239a0a07b43468bef4e599d988f37] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=36.1 K 2023-04-03 05:56:28,021 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting ac7f6381d2df42d8b113461c8b977958, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=270, earliestPutTs=1680501386418 2023-04-03 05:56:28,021 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 177a09b04bb24efe830292c62fa2c289, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=305, earliestPutTs=1680501386593 2023-04-03 05:56:28,021 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting c7c239a0a07b43468bef4e599d988f37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680501387229 2023-04-03 05:56:28,026 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#C#compaction#601 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:28,036 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/6ef1c6a79667490a837af5998043a833 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/6ef1c6a79667490a837af5998043a833 2023-04-03 05:56:28,039 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:28,039 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/C of 67f8cb6d380f875bf3bd14331600fad6 into 6ef1c6a79667490a837af5998043a833(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:28,039 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:28,039 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/C, priority=13, startTime=1680501387992; duration=0sec 2023-04-03 05:56:28,039 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:28,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:28,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:28,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:28,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:28,042 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:28,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:28,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:28,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:28,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,044 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:28,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:28,044 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,044 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:28,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,045 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:28,045 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:28,045 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:28,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:28,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:28,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:28,048 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:28,048 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,048 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:28,048 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:28,048 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-03 05:56:28,048 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:28,048 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:28,048 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:28,048 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:28,048 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:28,048 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:28,061 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403bf9665895f874cfd8ae2a34691c91dfc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403bf9665895f874cfd8ae2a34691c91dfc 2023-04-03 05:56:28,076 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=342, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/4834ab371fe9445eaaa566ae30bd521a 2023-04-03 05:56:28,092 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/3aac0100c0ff4d23a2376fdeafc8eb65 2023-04-03 05:56:28,105 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=342 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/3fcd13cbcf72418fb894618a4e55c412 2023-04-03 05:56:28,110 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/4834ab371fe9445eaaa566ae30bd521a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/4834ab371fe9445eaaa566ae30bd521a 2023-04-03 05:56:28,113 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/4834ab371fe9445eaaa566ae30bd521a, entries=150, sequenceid=342, filesize=25.4 K 2023-04-03 05:56:28,114 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/3aac0100c0ff4d23a2376fdeafc8eb65 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/3aac0100c0ff4d23a2376fdeafc8eb65 2023-04-03 05:56:28,117 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/3aac0100c0ff4d23a2376fdeafc8eb65, entries=150, sequenceid=342, filesize=11.9 K 2023-04-03 05:56:28,117 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/3fcd13cbcf72418fb894618a4e55c412 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/3fcd13cbcf72418fb894618a4e55c412 2023-04-03 05:56:28,120 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/3fcd13cbcf72418fb894618a4e55c412, entries=150, sequenceid=342, filesize=11.9 K 2023-04-03 05:56:28,121 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 67f8cb6d380f875bf3bd14331600fad6 in 73ms, sequenceid=342, compaction requested=false 2023-04-03 05:56:28,121 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:28,121 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:28,121 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:28,122 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,122 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,123 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,123 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:28,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e617c5f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:28,124 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:28,124 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,126 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:28,126 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:28,126 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,126 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:28,126 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:28,128 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:28,128 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:28,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:28,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:28,130 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:28,130 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:28,130 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:28,130 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:28,131 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:28,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:28,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,134 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:28,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,136 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:28,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:28,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:28,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:28,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:28,136 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,136 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:28,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:28,137 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,137 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:28,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:28,137 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,137 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:28,137 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:28,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:28,220 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:28,221 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:28,221 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:28,221 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:28,221 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:28,221 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:28,221 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:28,236 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040377b0f887a89f4f6b9667787d7818ea64 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040377b0f887a89f4f6b9667787d7818ea64 2023-04-03 05:56:28,237 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:28,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:28,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:28,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:28,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:28,242 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:28,242 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:28,242 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:28,242 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:28,244 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,244 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,244 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=352, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/38347733a8004ec182d6d97cbb438dbc 2023-04-03 05:56:28,246 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:28,246 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:28,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:28,246 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,246 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:28,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:28,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:28,247 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:28,247 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:28,247 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:28,247 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:28,248 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:28,248 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:28,248 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,248 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:28,248 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:28,248 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,248 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:28,248 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:28,248 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,248 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:28,249 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:28,251 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,251 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,252 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:28,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a9709a4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:28,253 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:28,254 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,254 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/ff5c58d1539e455cbd5ec35e487249bd 2023-04-03 05:56:28,255 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,255 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,255 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:28,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,256 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:28,256 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:28,256 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:28,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:28,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:28,258 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:28,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501388247, max timestamp=1680501388261Num uniques 859; 2023-04-03 05:56:28,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:28,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501388247, max timestamp=1680501388261Num uniques 859; 2023-04-03 05:56:28,263 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:28,264 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501388247, max timestamp=1680501388261Num uniques 859; 2023-04-03 05:56:28,264 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,266 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/7e59b21764a844ae84bdd5aee52124e7 2023-04-03 05:56:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:28,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:28,268 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:28,268 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:28,268 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:28,268 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:28,268 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:28,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/38347733a8004ec182d6d97cbb438dbc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/38347733a8004ec182d6d97cbb438dbc 2023-04-03 05:56:28,270 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,270 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,270 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:28,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,272 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,273 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/38347733a8004ec182d6d97cbb438dbc, entries=350, sequenceid=352, filesize=52.9 K 2023-04-03 05:56:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,274 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/ff5c58d1539e455cbd5ec35e487249bd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ff5c58d1539e455cbd5ec35e487249bd 2023-04-03 05:56:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ff5c58d1539e455cbd5ec35e487249bd, entries=150, sequenceid=352, filesize=11.9 K 2023-04-03 05:56:28,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/7e59b21764a844ae84bdd5aee52124e7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7e59b21764a844ae84bdd5aee52124e7 2023-04-03 05:56:28,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:28,280 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:28,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:28,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:28,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:28,281 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:28,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:28,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:28,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:28,281 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:28,281 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:28,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:28,283 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7e59b21764a844ae84bdd5aee52124e7, entries=150, sequenceid=352, filesize=11.9 K 2023-04-03 05:56:28,283 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=98.40 KB/100760 for 67f8cb6d380f875bf3bd14331600fad6 in 63ms, sequenceid=352, compaction requested=true 2023-04-03 05:56:28,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:28,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:28,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:28,283 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 05:56:28,283 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:28,284 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 106852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:28,284 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/A is initiating minor compaction (all files) 2023-04-03 05:56:28,284 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/A in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,285 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/1fa961ee8be844078903a81a9cd40ae1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/4834ab371fe9445eaaa566ae30bd521a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/38347733a8004ec182d6d97cbb438dbc] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=104.3 K 2023-04-03 05:56:28,285 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 1fa961ee8be844078903a81a9cd40ae1, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680501387229 2023-04-03 05:56:28,286 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 4834ab371fe9445eaaa566ae30bd521a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680501387563 2023-04-03 05:56:28,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:28,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=105.11 KB heapSize=295.53 KB 2023-04-03 05:56:28,286 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 38347733a8004ec182d6d97cbb438dbc, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680501388204 2023-04-03 05:56:28,286 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:28,286 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:28,286 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:28,286 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:28,286 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:28,286 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:28,307 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#A#compaction#609 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:28,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:28,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501448305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,313 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:28,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501448309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,320 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403d480de2cb203497e9df1b9943ef87682 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d480de2cb203497e9df1b9943ef87682 2023-04-03 05:56:28,381 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:28,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:28,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:28,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:28,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:28,385 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:28,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:28,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:28,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:28,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,387 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:28,387 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:28,387 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,387 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:28,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:28,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,388 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:28,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:28,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:28,388 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:28,388 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:28,388 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:28,388 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:28,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:28,389 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,389 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:28,389 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:28,389 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:28,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:28,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:28,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:28,390 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,390 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,391 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,391 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:28,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68eb3298[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:28,392 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:28,392 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,396 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,396 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:28,396 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,396 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:28,396 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:28,396 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:28,398 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,398 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:28,398 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,398 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:28,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:28,400 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:28,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:28,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:28,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:28,401 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:28,402 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,402 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,402 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:28,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,404 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,407 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:28,407 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:28,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:28,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:28,407 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:28,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:28,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:28,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:28,407 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,407 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,407 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,407 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:28,407 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:28,407 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,407 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,407 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,407 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:28,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:28,409 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/ee60214def5747759de3d4a433d69230 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ee60214def5747759de3d4a433d69230 2023-04-03 05:56:28,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:28,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501448410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,413 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/B of 67f8cb6d380f875bf3bd14331600fad6 into ee60214def5747759de3d4a433d69230(size=12.4 K), total size for store is 36.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:28,413 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:28,413 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/B, priority=13, startTime=1680501387992; duration=0sec 2023-04-03 05:56:28,413 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-03 05:56:28,413 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:28,414 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:28,414 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/C is initiating minor compaction (all files) 2023-04-03 05:56:28,414 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/C in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,414 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/6ef1c6a79667490a837af5998043a833, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/3fcd13cbcf72418fb894618a4e55c412, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7e59b21764a844ae84bdd5aee52124e7] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=36.2 K 2023-04-03 05:56:28,414 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6ef1c6a79667490a837af5998043a833, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680501387229 2023-04-03 05:56:28,415 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3fcd13cbcf72418fb894618a4e55c412, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680501387563 2023-04-03 05:56:28,415 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 7e59b21764a844ae84bdd5aee52124e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680501388209 2023-04-03 05:56:28,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:28,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501448414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,419 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#C#compaction#610 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:28,425 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/21878f72e5684efebe0e212ee618dc74 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/21878f72e5684efebe0e212ee618dc74 2023-04-03 05:56:28,428 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/C of 67f8cb6d380f875bf3bd14331600fad6 into 21878f72e5684efebe0e212ee618dc74(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:28,429 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:28,429 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/C, priority=13, startTime=1680501388283; duration=0sec 2023-04-03 05:56:28,429 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:28,429 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:28,431 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:28,431 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/B is initiating minor compaction (all files) 2023-04-03 05:56:28,431 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/B in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,432 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ee60214def5747759de3d4a433d69230, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/3aac0100c0ff4d23a2376fdeafc8eb65, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ff5c58d1539e455cbd5ec35e487249bd] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=36.2 K 2023-04-03 05:56:28,432 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ee60214def5747759de3d4a433d69230, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680501387229 2023-04-03 05:56:28,432 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3aac0100c0ff4d23a2376fdeafc8eb65, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=342, earliestPutTs=1680501387563 2023-04-03 05:56:28,432 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ff5c58d1539e455cbd5ec35e487249bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680501388209 2023-04-03 05:56:28,438 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#B#compaction#611 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:28,446 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/e858d4c0f59445e3beb2414c8c304775 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/e858d4c0f59445e3beb2414c8c304775 2023-04-03 05:56:28,450 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/B of 67f8cb6d380f875bf3bd14331600fad6 into e858d4c0f59445e3beb2414c8c304775(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:28,450 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:28,450 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/B, priority=13, startTime=1680501388283; duration=0sec 2023-04-03 05:56:28,450 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:28,507 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:28,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:28,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:28,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:28,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:28,510 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:28,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:28,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:28,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:28,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,514 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:28,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:28,514 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,514 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:28,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:28,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:28,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:28,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:28,515 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:28,515 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:28,515 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:28,515 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:28,515 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,515 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:28,515 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:28,516 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:28,516 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,516 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:28,516 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:28,516 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,516 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:28,516 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:28,517 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,517 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,517 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,517 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:28,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@342868f8[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:28,519 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:28,520 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,520 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,521 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:28,521 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:28,521 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:28,521 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,521 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:28,522 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,522 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:28,522 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:28,522 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:28,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:28,525 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:28,525 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:28,525 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:28,525 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:28,525 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:28,527 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,527 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,527 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,528 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,529 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:28,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:28,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:28,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:28,531 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:28,532 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:28,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:28,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:28,532 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:28,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:28,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:28,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:28,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501448614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,619 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:28,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501448617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,632 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:28,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:28,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:28,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:28,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:28,635 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:28,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:28,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:28,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:28,636 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,636 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,637 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,637 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:28,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:28,637 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,637 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:28,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:28,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:28,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:28,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:28,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:28,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:28,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:28,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:28,639 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:28,639 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,639 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:28,639 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:28,639 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,639 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:28,639 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:28,639 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,639 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:28,639 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:28,640 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,641 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,641 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:28,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@118e69a6[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:28,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:28,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,644 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:28,644 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,644 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:28,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:28,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:28,646 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,646 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,646 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:28,646 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:28,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:28,648 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:28,648 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:28,648 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:28,648 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:28,648 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:28,649 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,649 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,649 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:28,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:28,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:28,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:28,650 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,651 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:28,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:28,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:28,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:28,653 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:28,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:28,653 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:28,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:28,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:28,653 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:28,653 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:28,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:28,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:28,726 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=385, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/a6174f57cfe843438bb354ffc0465e88 2023-04-03 05:56:28,734 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/9d24ef73b9e14e649de21cdc7db043f0 2023-04-03 05:56:28,743 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=385 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/ce5ca5ae619c475ba16538ea2af8b454 2023-04-03 05:56:28,746 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/a6174f57cfe843438bb354ffc0465e88 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/a6174f57cfe843438bb354ffc0465e88 2023-04-03 05:56:28,749 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/a6174f57cfe843438bb354ffc0465e88, entries=250, sequenceid=385, filesize=39.2 K 2023-04-03 05:56:28,750 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/9d24ef73b9e14e649de21cdc7db043f0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9d24ef73b9e14e649de21cdc7db043f0 2023-04-03 05:56:28,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9d24ef73b9e14e649de21cdc7db043f0, entries=150, sequenceid=385, filesize=11.9 K 2023-04-03 05:56:28,753 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/ce5ca5ae619c475ba16538ea2af8b454 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ce5ca5ae619c475ba16538ea2af8b454 2023-04-03 05:56:28,754 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:28,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:28,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:28,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ce5ca5ae619c475ba16538ea2af8b454, entries=150, sequenceid=385, filesize=11.9 K 2023-04-03 05:56:28,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:28,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:28,757 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:28,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:28,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:28,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:28,757 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~111.82 KB/114500, heapSize ~315.41 KB/322976, currentSize=67.09 KB/68700 for 67f8cb6d380f875bf3bd14331600fad6 in 471ms, sequenceid=385, compaction requested=false 2023-04-03 05:56:28,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,757 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:28,757 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,759 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,759 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:28,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:28,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:28,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:28,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:28,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:28,761 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:28,761 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:28,761 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:28,761 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:28,761 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:28,761 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:28,761 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:28,761 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-03 05:56:28,761 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:28,761 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:28,761 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:28,762 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:28,762 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:28,762 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:28,775 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304033902931d8836425093df6be610b979f2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304033902931d8836425093df6be610b979f2 2023-04-03 05:56:28,781 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=401, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/d22f5287bcb4478681e307372cd8f9f0 2023-04-03 05:56:28,793 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/8dc3b4448c314871be536f68989d1f54 2023-04-03 05:56:28,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:28,921 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:28,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:28,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501448945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:28,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:28,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501448945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:29,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501449049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:29,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501449049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,135 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/417384cfc3504622b9e6a6c6f539fbc3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/417384cfc3504622b9e6a6c6f539fbc3 2023-04-03 05:56:29,139 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/A of 67f8cb6d380f875bf3bd14331600fad6 into 417384cfc3504622b9e6a6c6f539fbc3(size=26.1 K), total size for store is 65.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:29,139 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:29,140 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/A, priority=13, startTime=1680501388283; duration=0sec 2023-04-03 05:56:29,140 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:29,201 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/63309cb9e4704db3bbbc8df62c0a7ed8 2023-04-03 05:56:29,204 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/d22f5287bcb4478681e307372cd8f9f0 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d22f5287bcb4478681e307372cd8f9f0 2023-04-03 05:56:29,208 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d22f5287bcb4478681e307372cd8f9f0, entries=150, sequenceid=401, filesize=25.4 K 2023-04-03 05:56:29,208 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/8dc3b4448c314871be536f68989d1f54 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8dc3b4448c314871be536f68989d1f54 2023-04-03 05:56:29,211 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8dc3b4448c314871be536f68989d1f54, entries=150, sequenceid=401, filesize=11.9 K 2023-04-03 05:56:29,212 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/63309cb9e4704db3bbbc8df62c0a7ed8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/63309cb9e4704db3bbbc8df62c0a7ed8 2023-04-03 05:56:29,214 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/63309cb9e4704db3bbbc8df62c0a7ed8, entries=150, sequenceid=401, filesize=11.9 K 2023-04-03 05:56:29,215 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 67f8cb6d380f875bf3bd14331600fad6 in 454ms, sequenceid=401, compaction requested=true 2023-04-03 05:56:29,215 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:29,215 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,215 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:29,215 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:29,215 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,215 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:29,215 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:29,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,217 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:29,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1356981a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:29,218 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:29,219 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,220 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,220 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:29,220 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:29,220 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:29,220 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,220 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:29,224 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,224 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:29,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,224 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:29,224 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:29,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:29,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:29,226 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:29,226 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:29,226 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:29,226 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:29,226 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:29,228 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,228 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,228 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:29,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:29,228 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,232 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:29,232 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,232 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:29,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:29,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:29,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:29,232 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:29,233 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:29,232 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,233 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:29,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:29,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:29,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:29,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:29,254 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 05:56:29,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:29,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:29,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:29,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:29,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:29,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:29,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:29,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501449269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:29,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501449270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,333 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:29,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:29,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:29,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:29,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:29,336 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:29,336 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:29,336 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:29,336 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:29,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,337 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,338 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,338 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:29,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:29,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:29,338 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,338 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:29,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,339 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:29,339 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:29,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:29,340 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,340 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:29,340 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:29,340 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:29,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:29,342 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:29,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2298171a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:29,343 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:29,343 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,344 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,344 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,345 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:29,344 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:29,345 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:29,346 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,346 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:29,347 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:29,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:29,349 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:29,349 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:29,349 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:29,349 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:29,349 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:29,351 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,351 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,351 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:29,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:29,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:29,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,353 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:29,356 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:29,356 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:29,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:29,356 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:29,356 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:29,356 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,356 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,356 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,356 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:29,356 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:29,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:29,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:29,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:29,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501449373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,377 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:29,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501449375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,456 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:29,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:29,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:29,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:29,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:29,463 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:29,463 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:29,463 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:29,463 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:29,464 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,464 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,465 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:29,465 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:29,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:29,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,466 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:29,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,466 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:29,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:29,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:29,468 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:29,472 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:29,472 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:29,472 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:29,472 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:29,472 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:29,474 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,474 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,475 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,475 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:29,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5580c419[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:29,477 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:29,477 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,478 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:29,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:29,478 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:29,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:29,479 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,479 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:29,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:29,482 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:29,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:29,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:29,482 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:29,482 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:29,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,483 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:29,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,485 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,486 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:29,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:29,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:29,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:29,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:29,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:29,490 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:29,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,490 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:29,490 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:29,490 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:29,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:29,491 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,491 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,491 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:29,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501449576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:29,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501449578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,591 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:29,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:29,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:29,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:29,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:29,596 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:29,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:29,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:29,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:29,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,596 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,598 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:29,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:29,598 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,598 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,598 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:29,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:29,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:29,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:29,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:29,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:29,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:29,608 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:29,614 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:29,615 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:29,614 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,615 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:29,615 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:29,615 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,615 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:29,615 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:29,615 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,615 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:29,615 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:29,616 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:29,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37214456[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:29,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:29,620 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,622 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,622 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,622 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:29,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,628 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:29,628 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:29,628 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:29,635 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,635 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:29,636 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:29,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:29,639 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:29,639 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:29,639 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:29,639 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:29,639 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:29,640 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,640 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:29,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,640 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:29,641 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,646 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,646 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:29,646 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:29,646 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:29,646 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,647 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,647 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,647 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,647 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,647 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:29,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:29,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:29,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:29,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:29,647 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:29,647 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:29,667 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403e4202b1c89b04ec5b86d0ce29543c1f4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e4202b1c89b04ec5b86d0ce29543c1f4 2023-04-03 05:56:29,681 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=423, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/3b272021ed224360892ff45cff7a6789 2023-04-03 05:56:29,704 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/936656a29e68402a9c9f8498d510d430 2023-04-03 05:56:29,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/b506380221524b80bca3a0a2ba0b770e 2023-04-03 05:56:29,747 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:29,747 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/3b272021ed224360892ff45cff7a6789 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/3b272021ed224360892ff45cff7a6789 2023-04-03 05:56:29,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:29,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:29,752 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/3b272021ed224360892ff45cff7a6789, entries=150, sequenceid=423, filesize=25.4 K 2023-04-03 05:56:29,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:29,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:29,753 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:29,753 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:29,753 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:29,753 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:29,754 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,754 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,754 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/936656a29e68402a9c9f8498d510d430 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/936656a29e68402a9c9f8498d510d430 2023-04-03 05:56:29,756 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:29,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:29,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,757 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:29,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:29,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:29,757 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:29,758 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,758 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:29,758 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:29,758 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:29,758 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:29,758 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/936656a29e68402a9c9f8498d510d430, entries=150, sequenceid=423, filesize=11.9 K 2023-04-03 05:56:29,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/b506380221524b80bca3a0a2ba0b770e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b506380221524b80bca3a0a2ba0b770e 2023-04-03 05:56:29,759 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,759 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:29,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14af22a0[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:29,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:29,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:29,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,760 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:29,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:29,762 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,762 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:29,763 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:29,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b506380221524b80bca3a0a2ba0b770e, entries=150, sequenceid=423, filesize=11.9 K 2023-04-03 05:56:29,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,764 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=53.67 KB/54960 for 67f8cb6d380f875bf3bd14331600fad6 in 510ms, sequenceid=423, compaction requested=true 2023-04-03 05:56:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,764 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:29,764 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:29,764 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:29,764 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:29,765 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,768 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 118916 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:29,769 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/A is initiating minor compaction (all files) 2023-04-03 05:56:29,769 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/A in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,769 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/417384cfc3504622b9e6a6c6f539fbc3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/a6174f57cfe843438bb354ffc0465e88, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d22f5287bcb4478681e307372cd8f9f0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/3b272021ed224360892ff45cff7a6789] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=116.1 K 2023-04-03 05:56:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,769 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:29,769 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/B is initiating minor compaction (all files) 2023-04-03 05:56:29,769 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/B in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,769 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 417384cfc3504622b9e6a6c6f539fbc3, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680501388209 2023-04-03 05:56:29,769 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/e858d4c0f59445e3beb2414c8c304775, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9d24ef73b9e14e649de21cdc7db043f0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8dc3b4448c314871be536f68989d1f54, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/936656a29e68402a9c9f8498d510d430] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=48.2 K 2023-04-03 05:56:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,770 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a6174f57cfe843438bb354ffc0465e88, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680501388276 2023-04-03 05:56:29,770 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e858d4c0f59445e3beb2414c8c304775, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680501388209 2023-04-03 05:56:29,770 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting d22f5287bcb4478681e307372cd8f9f0, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680501388295 2023-04-03 05:56:29,770 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 9d24ef73b9e14e649de21cdc7db043f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680501388279 2023-04-03 05:56:29,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:29,770 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3b272021ed224360892ff45cff7a6789, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1680501388940 2023-04-03 05:56:29,771 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8dc3b4448c314871be536f68989d1f54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680501388295 2023-04-03 05:56:29,770 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:29,771 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:29,771 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:29,771 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:29,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:29,771 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:29,772 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 936656a29e68402a9c9f8498d510d430, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1680501388940 2023-04-03 05:56:29,774 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,774 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:29,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,775 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,777 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:29,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,786 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#B#compaction#621 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:29,787 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:29,787 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:29,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:29,788 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,788 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:29,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:29,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:29,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:29,788 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,788 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:29,788 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:29,788 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,788 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,788 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,788 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,796 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#A#compaction#620 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:29,820 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/8e5177a6fcd241b180a1c59ca7731d3e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8e5177a6fcd241b180a1c59ca7731d3e 2023-04-03 05:56:29,824 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/A of 67f8cb6d380f875bf3bd14331600fad6 into 8e5177a6fcd241b180a1c59ca7731d3e(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:29,824 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:29,824 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/A, priority=12, startTime=1680501389764; duration=0sec 2023-04-03 05:56:29,824 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:29,824 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:29,825 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49392 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:29,825 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/C is initiating minor compaction (all files) 2023-04-03 05:56:29,825 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/C in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,826 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/21878f72e5684efebe0e212ee618dc74, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ce5ca5ae619c475ba16538ea2af8b454, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/63309cb9e4704db3bbbc8df62c0a7ed8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b506380221524b80bca3a0a2ba0b770e] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=48.2 K 2023-04-03 05:56:29,826 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 21878f72e5684efebe0e212ee618dc74, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680501388209 2023-04-03 05:56:29,827 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ce5ca5ae619c475ba16538ea2af8b454, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=385, earliestPutTs=1680501388279 2023-04-03 05:56:29,827 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 63309cb9e4704db3bbbc8df62c0a7ed8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680501388295 2023-04-03 05:56:29,828 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b506380221524b80bca3a0a2ba0b770e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1680501388940 2023-04-03 05:56:29,842 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#C#compaction#622 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:29,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:29,884 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-03 05:56:29,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:29,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:29,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:29,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:29,884 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:29,884 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:29,888 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:29,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:29,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:29,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:29,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:29,892 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:29,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:29,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:29,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:29,893 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,893 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,894 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:29,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:29,894 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,895 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,895 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:29,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:29,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:29,896 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,896 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:29,896 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:29,896 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:29,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,897 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10108ce0[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:29,900 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:29,900 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,902 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:29,903 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:29,903 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:29,903 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,903 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:29,904 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,904 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:29,904 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,904 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,906 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040342024907404844e597b3f6071187588c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040342024907404844e597b3f6071187588c 2023-04-03 05:56:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:29,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:29,908 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:29,908 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:29,908 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:29,908 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:29,908 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:29,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,910 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:29,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:29,910 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,911 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:29,911 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:29,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:29,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:29,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:29,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:29,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:29,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:29,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:29,914 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:29,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:29,914 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:29,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,914 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:29,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:29,914 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:29,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:29,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:29,918 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=436, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/e6d00740bd9a451495f73234a1ec1219 2023-04-03 05:56:29,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:29,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501449916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:29,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:29,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501449917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,014 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:30,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:30,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:30,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:30,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:30,020 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:30,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:30,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:30,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:30,023 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501450021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:30,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501450021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:30,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,025 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:30,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:30,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:30,025 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,025 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:30,025 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:30,025 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:30,026 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:30,026 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,027 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,027 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:30,027 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@658d09f6[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:30,027 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:30,027 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:30,027 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,029 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,029 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:30,029 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,029 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,030 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:30,030 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:30,030 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:30,030 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:30,031 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,031 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,031 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,031 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:30,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:30,036 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:30,036 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:30,036 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:30,036 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:30,036 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:30,039 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,039 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,039 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,040 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,041 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,047 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,047 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,047 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,047 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:30,047 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:30,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:30,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,047 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,047 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,047 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,047 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,047 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,047 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:30,048 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:30,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,047 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,148 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:30,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:30,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:30,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:30,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:30,150 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:30,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:30,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:30,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,152 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,153 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,153 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:30,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:30,153 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:30,153 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:30,154 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,154 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:30,154 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:30,154 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:30,154 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:30,155 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,155 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:30,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7252693f[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:30,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:30,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,159 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,159 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:30,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,159 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:30,159 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:30,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:30,161 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:30,161 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,161 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,161 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:30,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:30,163 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:30,163 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:30,163 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:30,163 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:30,163 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:30,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,165 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,166 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:30,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:30,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:30,168 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,168 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:30,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,168 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:30,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,201 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/874422390c0648f5ad33ec35fc2fa0e6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/874422390c0648f5ad33ec35fc2fa0e6 2023-04-03 05:56:30,205 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/B of 67f8cb6d380f875bf3bd14331600fad6 into 874422390c0648f5ad33ec35fc2fa0e6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:30,205 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:30,205 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/B, priority=12, startTime=1680501389764; duration=0sec 2023-04-03 05:56:30,205 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:30,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:30,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501450225, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:30,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501450227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,268 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:30,269 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/7029e430805047fbb154036dd23bbfb6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7029e430805047fbb154036dd23bbfb6 2023-04-03 05:56:30,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:30,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:30,273 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/C of 67f8cb6d380f875bf3bd14331600fad6 into 7029e430805047fbb154036dd23bbfb6(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:30,273 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:30,273 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/C, priority=12, startTime=1680501389765; duration=0sec 2023-04-03 05:56:30,274 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:30,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:30,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:30,274 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:30,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:30,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:30,275 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,275 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,280 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:30,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:30,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:30,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:30,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:30,284 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:30,284 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,284 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:30,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:30,290 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,290 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:30,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:30,290 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:30,290 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:30,290 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:30,291 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,291 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:30,291 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:30,292 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,292 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,292 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,292 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:30,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2121cb60[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:30,294 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:30,294 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,295 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,295 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:30,295 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,295 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:30,295 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:30,295 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:30,297 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,297 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:30,297 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,297 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:30,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:30,299 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:30,299 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:30,299 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:30,299 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:30,299 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:30,302 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,302 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,302 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,303 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,304 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,306 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,306 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,306 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:30,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:30,306 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:30,307 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:30,307 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:30,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,306 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,307 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,307 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,307 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,307 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,307 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,338 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/1af4ed1f1cdb4a3d979c1105026f5259 2023-04-03 05:56:30,407 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:30,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:30,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:30,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:30,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:30,412 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:30,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:30,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:30,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,412 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,414 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,414 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,414 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,414 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:30,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:30,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:30,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:30,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:30,415 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,415 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:30,415 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:30,415 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:30,415 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:30,417 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,417 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:30,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c9f28e6[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:30,418 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:30,418 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,421 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:30,421 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:30,421 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,421 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,422 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,422 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:30,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:30,424 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:30,424 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:30,424 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:30,424 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:30,424 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:30,426 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,426 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,426 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,426 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,427 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,430 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,430 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,430 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:30,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:30,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,430 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:30,430 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,430 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:30,430 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:30,430 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,430 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,430 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,430 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,430 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,430 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:30,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501450529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,530 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:30,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:30,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501450530, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:30,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:30,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:30,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:30,533 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:30,534 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,534 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:30,534 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:30,534 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,534 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,540 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,540 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,540 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,540 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:30,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:30,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,541 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:30,541 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:30,541 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:30,542 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,542 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:30,548 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:30,548 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:30,548 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,548 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:30,548 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:30,548 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,548 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:30,548 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:30,548 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,549 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:30,549 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:30,550 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,550 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,550 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,550 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:30,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:30,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@87cd019[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:30,552 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:30,552 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,553 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,553 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:30,553 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,553 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:30,553 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:30,553 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:30,555 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,556 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:30,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:30,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:30,569 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:30,569 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:30,569 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:30,569 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:30,569 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:30,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,572 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,572 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,574 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,576 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:30,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:30,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:30,576 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:30,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,576 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:30,576 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,577 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,677 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:30,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:30,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:30,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:30,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:30,679 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:30,679 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,679 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:30,679 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:30,680 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,680 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,681 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,681 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,681 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,681 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:30,682 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:30,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:30,684 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:30,684 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:30,684 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:30,684 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,684 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:30,692 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:30,692 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:30,692 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,692 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:30,692 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:30,692 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,692 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:30,692 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:30,692 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,692 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:30,692 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:30,693 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,693 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,694 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:30,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:30,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69864bb8[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:30,695 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:30,695 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,698 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:30,698 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:30,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,698 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:30,700 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,700 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:30,700 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,700 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:30,708 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:30,708 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:30,708 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:30,708 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:30,708 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:30,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,711 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,712 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:30,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:30,715 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,716 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,716 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:30,716 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:30,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:30,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,754 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/0fab22a510f04530855196b53fadc8c8 2023-04-03 05:56:30,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/e6d00740bd9a451495f73234a1ec1219 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e6d00740bd9a451495f73234a1ec1219 2023-04-03 05:56:30,762 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e6d00740bd9a451495f73234a1ec1219, entries=150, sequenceid=436, filesize=25.4 K 2023-04-03 05:56:30,762 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/1af4ed1f1cdb4a3d979c1105026f5259 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1af4ed1f1cdb4a3d979c1105026f5259 2023-04-03 05:56:30,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1af4ed1f1cdb4a3d979c1105026f5259, entries=150, sequenceid=436, filesize=11.9 K 2023-04-03 05:56:30,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/0fab22a510f04530855196b53fadc8c8 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/0fab22a510f04530855196b53fadc8c8 2023-04-03 05:56:30,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/0fab22a510f04530855196b53fadc8c8, entries=150, sequenceid=436, filesize=11.9 K 2023-04-03 05:56:30,772 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 67f8cb6d380f875bf3bd14331600fad6 in 888ms, sequenceid=436, compaction requested=false 2023-04-03 05:56:30,772 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:30,816 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:30,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:30,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:30,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:30,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:30,832 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:30,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:30,832 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:30,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,833 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,835 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,835 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,835 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,835 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:30,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,836 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,836 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:30,836 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:30,837 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:30,837 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:30,837 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:30,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:30,840 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:30,840 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,840 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:30,843 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-03 05:56:30,844 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:30,844 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:30,844 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:30,844 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:30,844 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:30,844 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:30,859 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403447d2c30a6d645a6adf94ce3538894f2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403447d2c30a6d645a6adf94ce3538894f2 2023-04-03 05:56:30,877 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=458, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/ab23093bec2f4166b571837c9a5d8e92 2023-04-03 05:56:30,889 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/6d6b1eef929a4d0d9c54919a58209899 2023-04-03 05:56:30,906 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/ecc113b745d04dde949be56e34450780 2023-04-03 05:56:30,909 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/ab23093bec2f4166b571837c9a5d8e92 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/ab23093bec2f4166b571837c9a5d8e92 2023-04-03 05:56:30,912 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/ab23093bec2f4166b571837c9a5d8e92, entries=150, sequenceid=458, filesize=25.4 K 2023-04-03 05:56:30,913 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/6d6b1eef929a4d0d9c54919a58209899 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6d6b1eef929a4d0d9c54919a58209899 2023-04-03 05:56:30,916 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6d6b1eef929a4d0d9c54919a58209899, entries=150, sequenceid=458, filesize=11.9 K 2023-04-03 05:56:30,917 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/ecc113b745d04dde949be56e34450780 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ecc113b745d04dde949be56e34450780 2023-04-03 05:56:30,920 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ecc113b745d04dde949be56e34450780, entries=150, sequenceid=458, filesize=11.9 K 2023-04-03 05:56:30,920 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for 67f8cb6d380f875bf3bd14331600fad6 in 77ms, sequenceid=458, compaction requested=true 2023-04-03 05:56:30,920 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:30,920 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:30,921 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:30,922 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,922 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:30,924 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4653f723[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:30,924 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:30,924 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,925 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,925 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:30,925 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:30,925 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:30,925 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,926 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:30,927 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:30,927 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,927 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,927 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:30,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:30,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:30,929 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:30,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:30,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:30,929 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:30,929 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:30,932 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,932 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,932 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:30,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:30,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:30,933 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:30,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,934 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:30,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:30,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:30,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:30,937 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:30,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:30,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:30,937 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:30,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:30,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:30,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:30,937 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:30,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:30,937 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:30,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:30,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:30,937 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,938 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:30,938 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,038 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:31,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:31,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:31,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:31,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:31,041 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:31,041 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:31,041 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:31,041 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:31,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,043 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,043 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:31,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:31,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:31,043 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,043 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:31,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:31,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:31,044 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:31,044 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:31,044 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:31,044 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:31,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:31,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:31,045 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:31,045 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:31,045 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:31,045 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=6.71 KB heapSize=20.67 KB 2023-04-03 05:56:31,045 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:31,045 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:31,046 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:31,046 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:31,046 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:31,046 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:31,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:31,066 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:31,068 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304036509c97cfe354f21b941a0cbf8f47e1b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036509c97cfe354f21b941a0cbf8f47e1b 2023-04-03 05:56:31,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501391072, max timestamp=1680501391084Num uniques 859; 2023-04-03 05:56:31,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:31,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501391072, max timestamp=1680501391084Num uniques 859; 2023-04-03 05:56:31,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:31,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501391072, max timestamp=1680501391084Num uniques 859; 2023-04-03 05:56:31,086 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:31,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680501391099, max timestamp=1680501391110Num uniques 1018; 2023-04-03 05:56:31,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-03 05:56:31,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680501391099, max timestamp=1680501391110Num uniques 1018; 2023-04-03 05:56:31,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680501391099, max timestamp=1680501391110Num uniques 1018; 2023-04-03 05:56:31,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-03 05:56:31,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-03 05:56:31,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680501391129, max timestamp=1680501391138Num uniques 1077; 2023-04-03 05:56:31,140 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-03 05:56:31,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680501391129, max timestamp=1680501391138Num uniques 1077; 2023-04-03 05:56:31,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-03 05:56:31,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680501391129, max timestamp=1680501391138Num uniques 1077; 2023-04-03 05:56:31,141 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49322, new segment data size=8102, suffix heap size=139976,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1077, new segment cells count=177 2023-04-03 05:56:31,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680501391139, max timestamp=1680501391161Num uniques 1036; 2023-04-03 05:56:31,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-03 05:56:31,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680501391139, max timestamp=1680501391161Num uniques 1036; 2023-04-03 05:56:31,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1680501391139, max timestamp=1680501391161Num uniques 1036; 2023-04-03 05:56:31,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-03 05:56:31,163 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=10806, suffix heap size=135712,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=236 2023-04-03 05:56:31,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680501391178, max timestamp=1680501391185Num uniques 1095; 2023-04-03 05:56:31,187 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-03 05:56:31,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680501391178, max timestamp=1680501391185Num uniques 1095; 2023-04-03 05:56:31,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-03 05:56:31,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680501391178, max timestamp=1680501391185Num uniques 1095; 2023-04-03 05:56:31,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-04-03 05:56:31,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680501391207, max timestamp=1680501391214Num uniques 1054; 2023-04-03 05:56:31,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-03 05:56:31,216 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680501391207, max timestamp=1680501391214Num uniques 1054; 2023-04-03 05:56:31,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-03 05:56:31,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680501391207, max timestamp=1680501391214Num uniques 1054; 2023-04-03 05:56:31,217 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-03 05:56:31,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680501391233, max timestamp=1680501391242Num uniques 1063; 2023-04-03 05:56:31,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-03 05:56:31,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680501391233, max timestamp=1680501391242Num uniques 1063; 2023-04-03 05:56:31,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-03 05:56:31,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680501391233, max timestamp=1680501391242Num uniques 1063; 2023-04-03 05:56:31,245 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=9748, suffix heap size=138520,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=213 2023-04-03 05:56:31,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680501391259, max timestamp=1680501391278Num uniques 1072; 2023-04-03 05:56:31,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-03 05:56:31,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680501391259, max timestamp=1680501391278Num uniques 1072; 2023-04-03 05:56:31,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680501391259, max timestamp=1680501391278Num uniques 1072; 2023-04-03 05:56:31,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-03 05:56:31,281 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-03 05:56:31,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680501391301, max timestamp=1680501391311Num uniques 1031; 2023-04-03 05:56:31,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680501391301, max timestamp=1680501391311Num uniques 1031; 2023-04-03 05:56:31,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-03 05:56:31,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-03 05:56:31,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680501391301, max timestamp=1680501391311Num uniques 1031; 2023-04-03 05:56:31,314 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-03 05:56:31,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680501391327, max timestamp=1680501391340Num uniques 1040; 2023-04-03 05:56:31,342 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-03 05:56:31,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680501391327, max timestamp=1680501391340Num uniques 1040; 2023-04-03 05:56:31,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-03 05:56:31,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=190, cellSize=8700, totalHeapSize=20000, min timestamp=1680501391327, max timestamp=1680501391340Num uniques 1040; 2023-04-03 05:56:31,343 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=8700, suffix heap size=136128,new segment heap size=20000 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=190 2023-04-03 05:56:31,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680501391361, max timestamp=1680501391369Num uniques 1049; 2023-04-03 05:56:31,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680501391361, max timestamp=1680501391369Num uniques 1049; 2023-04-03 05:56:31,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-03 05:56:31,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680501391361, max timestamp=1680501391369Num uniques 1049; 2023-04-03 05:56:31,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-03 05:56:31,372 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48044, new segment data size=9114, suffix heap size=137064,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1049, new segment cells count=199 2023-04-03 05:56:31,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680501391388, max timestamp=1680501391396Num uniques 1058; 2023-04-03 05:56:31,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-03 05:56:31,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680501391388, max timestamp=1680501391396Num uniques 1058; 2023-04-03 05:56:31,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-03 05:56:31,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680501391388, max timestamp=1680501391396Num uniques 1058; 2023-04-03 05:56:31,399 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-03 05:56:31,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680501391411, max timestamp=1680501391419Num uniques 1017; 2023-04-03 05:56:31,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-03 05:56:31,421 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680501391411, max timestamp=1680501391419Num uniques 1017; 2023-04-03 05:56:31,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-03 05:56:31,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1680501391411, max timestamp=1680501391419Num uniques 1017; 2023-04-03 05:56:31,422 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=9932, suffix heap size=133736,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=217 2023-04-03 05:56:31,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680501391432, max timestamp=1680501391441Num uniques 1076; 2023-04-03 05:56:31,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680501391432, max timestamp=1680501391441Num uniques 1076; 2023-04-03 05:56:31,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-03 05:56:31,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=176, cellSize=8056, totalHeapSize=18544, min timestamp=1680501391432, max timestamp=1680501391441Num uniques 1076; 2023-04-03 05:56:31,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-03 05:56:31,444 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=8056, suffix heap size=139872,new segment heap size=18544 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=176 2023-04-03 05:56:31,473 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=462, memsize=2.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/f1d63db8586746bd8d2b4792cebf8c0b 2023-04-03 05:56:31,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680501391466, max timestamp=1680501391474Num uniques 1035; 2023-04-03 05:56:31,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-03 05:56:31,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680501391466, max timestamp=1680501391474Num uniques 1035; 2023-04-03 05:56:31,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-03 05:56:31,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680501391466, max timestamp=1680501391474Num uniques 1035; 2023-04-03 05:56:31,477 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47400, new segment data size=10760, suffix heap size=135608,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1035, new segment cells count=235 2023-04-03 05:56:31,500 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/010b2aa17d1b4468b246c0e3b0cf5bcd 2023-04-03 05:56:31,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680501391496, max timestamp=1680501391508Num uniques 1094; 2023-04-03 05:56:31,511 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-03 05:56:31,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680501391496, max timestamp=1680501391508Num uniques 1094; 2023-04-03 05:56:31,512 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-03 05:56:31,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=194, cellSize=8884, totalHeapSize=20416, min timestamp=1680501391496, max timestamp=1680501391508Num uniques 1094; 2023-04-03 05:56:31,514 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=8884, suffix heap size=141744,new segment heap size=20416 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=194 2023-04-03 05:56:31,514 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.24 KB at sequenceid=462 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/0df77ba5171e4bc58b972059d3ed1d77 2023-04-03 05:56:31,518 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/f1d63db8586746bd8d2b4792cebf8c0b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/f1d63db8586746bd8d2b4792cebf8c0b 2023-04-03 05:56:31,521 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/f1d63db8586746bd8d2b4792cebf8c0b, entries=50, sequenceid=462, filesize=11.7 K 2023-04-03 05:56:31,521 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/010b2aa17d1b4468b246c0e3b0cf5bcd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/010b2aa17d1b4468b246c0e3b0cf5bcd 2023-04-03 05:56:31,524 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/010b2aa17d1b4468b246c0e3b0cf5bcd, entries=50, sequenceid=462, filesize=7.1 K 2023-04-03 05:56:31,525 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/0df77ba5171e4bc58b972059d3ed1d77 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/0df77ba5171e4bc58b972059d3ed1d77 2023-04-03 05:56:31,528 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/0df77ba5171e4bc58b972059d3ed1d77, entries=50, sequenceid=462, filesize=7.1 K 2023-04-03 05:56:31,528 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~6.71 KB/6870, heapSize ~20.63 KB/21120, currentSize=80.51 KB/82440 for 67f8cb6d380f875bf3bd14331600fad6 in 483ms, sequenceid=462, compaction requested=true 2023-04-03 05:56:31,528 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:31,528 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:31,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:31,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:31,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:31,529 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:31,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:31,530 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=87.22 KB heapSize=242.25 KB 2023-04-03 05:56:31,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,530 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:31,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:31,530 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:31,531 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:31,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:31,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:31,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:31,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:31,531 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:31,531 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:31,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@236e2f17[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:31,532 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:31,532 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,543 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,543 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:31,543 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:31,543 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:31,544 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,544 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:31,545 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,545 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:31,545 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:31,545 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:31,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:31,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:31,547 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:31,547 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:31,548 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:31,548 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:31,548 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:31,548 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:31,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,549 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:31,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:31,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:31,550 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:31,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,552 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,554 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,554 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:31,554 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:31,554 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:31,554 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,554 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:31,554 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:31,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:31,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:31,554 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,555 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,555 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,555 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:31,555 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:31,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:31,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:31,565 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:31,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 404 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501451561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,565 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:31,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 421 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501451562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,655 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:31,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:31,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:31,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:31,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:31,658 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:31,658 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:31,658 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:31,658 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:31,658 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,658 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,660 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:31,660 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:31,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:31,661 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,661 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,661 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:31,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,661 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:31,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:31,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:31,662 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:31,662 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:31,662 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:31,662 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:31,662 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:31,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,664 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:31,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,664 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:31,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:31,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dfccf7b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:31,666 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:31,666 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,667 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,667 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:31,667 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:31,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,667 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:31,667 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:31,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:31,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 406 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501451666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:31,668 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 423 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501451666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,668 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:31,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:31,668 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:31,668 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:31,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:31,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:31,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:31,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:31,671 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:31,671 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:31,671 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:31,671 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:31,671 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:31,673 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,673 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:31,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,673 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:31,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:31,673 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,674 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,676 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:31,676 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,676 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:31,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:31,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:31,676 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:31,676 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,677 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:31,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:31,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:31,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:31,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:31,677 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,677 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:31,677 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:31,677 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,677 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,677 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,677 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,777 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:31,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:31,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:31,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:31,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:31,780 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:31,780 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:31,780 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:31,780 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:31,780 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,781 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,782 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,782 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:31,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:31,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:31,782 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,782 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:31,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:31,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:31,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:31,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:31,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:31,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:31,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:31,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:31,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:31,783 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:31,783 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:31,784 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:31,784 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:31,784 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:31,784 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:31,784 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,784 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:31,784 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:31,785 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,785 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:31,785 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,785 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:31,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:31,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:31,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@167c96f7[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:31,787 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:31,787 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,789 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,789 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:31,789 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:31,789 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:31,789 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,789 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:31,790 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:31,790 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:31,790 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,790 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:31,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:31,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:31,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:31,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:31,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:31,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:31,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:31,793 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:31,794 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:31,794 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:31,794 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:31,794 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:31,795 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,795 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:31,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,795 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:31,796 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:31,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,797 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:31,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,799 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,799 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,799 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,799 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,799 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:31,799 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:31,799 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:31,799 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,799 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:31,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:31,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:31,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:31,799 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:31,800 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:31,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:31,800 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:31,800 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:31,870 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:31,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 425 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501451869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:31,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 408 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501451870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,900 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:31,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:31,902 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:31,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:31,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:31,903 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:31,903 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:31,903 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:31,903 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:31,904 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,904 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,906 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,906 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:31,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:31,907 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,907 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,907 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:31,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:31,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:31,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:31,908 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:31,908 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:31,908 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:31,908 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:31,908 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,909 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:31,909 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:31,910 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:31,910 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:31,911 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,911 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:31,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:31,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fcf91f4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:31,912 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:31,912 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,913 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,913 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,913 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:31,913 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:31,913 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:31,913 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:31,918 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:31,918 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:31,918 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,918 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:31,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:31,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:31,920 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:31,921 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:31,921 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:31,921 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:31,921 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:31,922 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,922 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:31,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,922 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:31,922 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:31,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,924 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:31,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:31,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,925 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,925 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,925 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:31,926 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:31,926 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,926 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:31,925 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:31,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:31,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:31,926 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:31,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:31,926 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:31,926 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:31,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:31,926 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:31,926 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:31,926 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:31,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:31,948 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040326dad49441564d9588bc449a19af8915 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040326dad49441564d9588bc449a19af8915 2023-04-03 05:56:31,956 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=750, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/07f0c76dd30249898ca4e5089ecab4b4 2023-04-03 05:56:31,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=750 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/6921680f9dda4de2879cc425ef1c8496 2023-04-03 05:56:32,026 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:32,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:32,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:32,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:32,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:32,029 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:32,029 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:32,029 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:32,029 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:32,030 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,030 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,031 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:32,031 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:32,031 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,031 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:32,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:32,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:32,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:32,032 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:32,033 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:32,033 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:32,033 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:32,033 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:32,033 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:32,033 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:32,033 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:32,033 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,033 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:32,033 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:32,035 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,036 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:32,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:32,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:32,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d8b198d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:32,037 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:32,037 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,038 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:32,038 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:32,039 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,039 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:32,039 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,040 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:32,040 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:32,040 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:32,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:32,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:32,042 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:32,042 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:32,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:32,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:32,042 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:32,042 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:32,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:32,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,044 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:32,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:32,044 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,045 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:32,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,052 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:32,052 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:32,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:32,053 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:32,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:32,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:32,053 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:32,053 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:32,052 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:32,053 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,053 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,053 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,053 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,053 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:32,053 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:32,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:32,153 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:32,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:32,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:32,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:32,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:32,156 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:32,156 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:32,156 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:32,156 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:32,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,157 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,158 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:32,158 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:32,158 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,158 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:32,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:32,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:32,159 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:32,159 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:32,160 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:32,160 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:32,160 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:32,160 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:32,160 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,162 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:32,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,162 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:32,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:32,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36a78a50[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:32,163 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:32,163 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,164 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,165 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,165 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:32,165 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:32,165 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:32,165 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:32,168 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:32,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:32,168 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:32,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:32,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:32,170 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:32,170 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:32,170 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:32,170 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:32,170 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:32,171 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,171 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,171 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:32,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 427 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501452171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,172 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:32,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,172 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:32,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,173 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,175 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,175 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:32,176 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:32,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:32,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:32,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:32,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:32,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:32,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,176 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,176 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:32,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:32,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:32,176 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:32,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:32,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 410 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501452175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,276 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:32,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:32,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:32,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:32,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:32,279 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:32,279 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:32,279 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:32,279 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:32,280 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,280 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,281 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:32,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:32,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,281 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:32,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:32,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,282 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:32,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:32,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:32,282 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:32,282 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:32,282 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:32,282 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:32,282 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:32,282 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:32,282 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:32,282 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:32,283 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:32,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:32,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:32,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:32,283 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:32,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,285 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:32,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,285 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:32,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:32,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:32,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@449e795d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:32,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:32,287 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,288 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:32,288 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:32,288 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,288 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:32,289 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:32,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:32,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:32,289 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:32,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:32,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:32,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:32,291 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:32,291 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:32,292 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:32,292 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:32,292 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:32,292 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:32,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:32,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,293 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:32,293 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:32,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:32,294 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:32,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:32,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:32,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:32,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:32,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:32,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:32,303 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:32,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:32,303 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:32,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:32,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:32,304 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:32,304 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:32,372 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=750 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/e92ba63c5067418b9bd2059a0b22a38f 2023-04-03 05:56:32,375 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/07f0c76dd30249898ca4e5089ecab4b4 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/07f0c76dd30249898ca4e5089ecab4b4 2023-04-03 05:56:32,378 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/07f0c76dd30249898ca4e5089ecab4b4, entries=250, sequenceid=750, filesize=39.2 K 2023-04-03 05:56:32,379 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/6921680f9dda4de2879cc425ef1c8496 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6921680f9dda4de2879cc425ef1c8496 2023-04-03 05:56:32,382 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6921680f9dda4de2879cc425ef1c8496, entries=150, sequenceid=750, filesize=11.9 K 2023-04-03 05:56:32,382 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/e92ba63c5067418b9bd2059a0b22a38f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e92ba63c5067418b9bd2059a0b22a38f 2023-04-03 05:56:32,385 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e92ba63c5067418b9bd2059a0b22a38f, entries=150, sequenceid=750, filesize=11.9 K 2023-04-03 05:56:32,385 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~262.13 KB/268416, currentSize=87.22 KB/89310 for 67f8cb6d380f875bf3bd14331600fad6 in 855ms, sequenceid=750, compaction requested=true 2023-04-03 05:56:32,385 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:32,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:32,386 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:56:32,386 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:56:32,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:32,386 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:32,387 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 131008 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:56:32,387 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56828 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:56:32,387 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/A is initiating minor compaction (all files) 2023-04-03 05:56:32,387 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/B is initiating minor compaction (all files) 2023-04-03 05:56:32,387 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/A in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:32,387 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/B in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:32,387 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8e5177a6fcd241b180a1c59ca7731d3e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e6d00740bd9a451495f73234a1ec1219, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/ab23093bec2f4166b571837c9a5d8e92, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/f1d63db8586746bd8d2b4792cebf8c0b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/07f0c76dd30249898ca4e5089ecab4b4] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=127.9 K 2023-04-03 05:56:32,387 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/874422390c0648f5ad33ec35fc2fa0e6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1af4ed1f1cdb4a3d979c1105026f5259, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6d6b1eef929a4d0d9c54919a58209899, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/010b2aa17d1b4468b246c0e3b0cf5bcd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6921680f9dda4de2879cc425ef1c8496] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=55.5 K 2023-04-03 05:56:32,387 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8e5177a6fcd241b180a1c59ca7731d3e, keycount=150, bloomtype=ROW, size=26.2 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1680501388940 2023-04-03 05:56:32,388 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 874422390c0648f5ad33ec35fc2fa0e6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1680501388940 2023-04-03 05:56:32,388 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting e6d00740bd9a451495f73234a1ec1219, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680501389263 2023-04-03 05:56:32,388 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1af4ed1f1cdb4a3d979c1105026f5259, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680501389263 2023-04-03 05:56:32,388 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting ab23093bec2f4166b571837c9a5d8e92, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680501389908 2023-04-03 05:56:32,388 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6d6b1eef929a4d0d9c54919a58209899, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680501389908 2023-04-03 05:56:32,388 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting f1d63db8586746bd8d2b4792cebf8c0b, keycount=50, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1680501391044 2023-04-03 05:56:32,389 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 010b2aa17d1b4468b246c0e3b0cf5bcd, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1680501391044 2023-04-03 05:56:32,389 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 07f0c76dd30249898ca4e5089ecab4b4, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680501391517 2023-04-03 05:56:32,389 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6921680f9dda4de2879cc425ef1c8496, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680501391527 2023-04-03 05:56:32,397 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#B#compaction#635 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:32,398 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#A#compaction#636 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:32,404 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:32,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:32,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:32,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:32,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:32,408 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:32,408 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:32,408 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:32,408 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:32,409 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,409 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,411 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:32,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:32,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,411 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:32,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:32,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:32,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:32,412 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:32,412 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:32,412 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:32,412 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:32,413 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:32,413 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:32,413 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:32,413 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:32,413 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-03 05:56:32,413 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:32,413 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:32,413 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:32,413 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:32,413 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:32,413 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:32,414 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/5ffce3f0d7d0473d8b4adf787e6ca699 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/5ffce3f0d7d0473d8b4adf787e6ca699 2023-04-03 05:56:32,418 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/B of 67f8cb6d380f875bf3bd14331600fad6 into 5ffce3f0d7d0473d8b4adf787e6ca699(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:32,418 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:32,418 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/B, priority=11, startTime=1680501392386; duration=0sec 2023-04-03 05:56:32,418 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:32,418 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-03 05:56:32,420 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 56828 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-03 05:56:32,420 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/C is initiating minor compaction (all files) 2023-04-03 05:56:32,420 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/C in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:32,420 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7029e430805047fbb154036dd23bbfb6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/0fab22a510f04530855196b53fadc8c8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ecc113b745d04dde949be56e34450780, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/0df77ba5171e4bc58b972059d3ed1d77, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e92ba63c5067418b9bd2059a0b22a38f] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=55.5 K 2023-04-03 05:56:32,421 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 7029e430805047fbb154036dd23bbfb6, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1680501388940 2023-04-03 05:56:32,421 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0fab22a510f04530855196b53fadc8c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680501389263 2023-04-03 05:56:32,422 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ecc113b745d04dde949be56e34450780, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680501389908 2023-04-03 05:56:32,422 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 0df77ba5171e4bc58b972059d3ed1d77, keycount=50, bloomtype=ROW, size=7.1 K, encoding=NONE, compression=NONE, seqNum=462, earliestPutTs=1680501391044 2023-04-03 05:56:32,422 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting e92ba63c5067418b9bd2059a0b22a38f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680501391527 2023-04-03 05:56:32,428 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040386ec2c3b422442fd9f9cff032a8f0d8d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040386ec2c3b422442fd9f9cff032a8f0d8d 2023-04-03 05:56:32,429 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#C#compaction#638 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:32,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:32,680 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:32,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:32,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 418 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501452695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,698 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:32,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 436 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501452696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:32,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 420 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501452797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:32,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 438 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501452799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:32,834 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=766, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/8893b2510c504aa7ab34914625316caf 2023-04-03 05:56:32,836 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/ca7207bc83d04712b6fe17a50abdaa34 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ca7207bc83d04712b6fe17a50abdaa34 2023-04-03 05:56:32,840 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/C of 67f8cb6d380f875bf3bd14331600fad6 into ca7207bc83d04712b6fe17a50abdaa34(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:32,840 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:32,840 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/C, priority=11, startTime=1680501392386; duration=0sec 2023-04-03 05:56:32,840 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:33,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:33,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 422 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501453001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:33,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 440 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501453004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,222 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/15fd6f7755894e448849981db4dd97ad as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/15fd6f7755894e448849981db4dd97ad 2023-04-03 05:56:33,226 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/A of 67f8cb6d380f875bf3bd14331600fad6 into 15fd6f7755894e448849981db4dd97ad(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:33,226 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:33,226 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/A, priority=11, startTime=1680501392386; duration=0sec 2023-04-03 05:56:33,226 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:33,243 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/01ce0b6db23c4d8aaf637e95dc59d83b 2023-04-03 05:56:33,250 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/272772f5653348a694cd1f3ff2496e52 2023-04-03 05:56:33,253 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/8893b2510c504aa7ab34914625316caf as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8893b2510c504aa7ab34914625316caf 2023-04-03 05:56:33,256 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8893b2510c504aa7ab34914625316caf, entries=150, sequenceid=766, filesize=25.4 K 2023-04-03 05:56:33,257 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/01ce0b6db23c4d8aaf637e95dc59d83b as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/01ce0b6db23c4d8aaf637e95dc59d83b 2023-04-03 05:56:33,259 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/01ce0b6db23c4d8aaf637e95dc59d83b, entries=150, sequenceid=766, filesize=11.9 K 2023-04-03 05:56:33,260 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/272772f5653348a694cd1f3ff2496e52 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/272772f5653348a694cd1f3ff2496e52 2023-04-03 05:56:33,263 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/272772f5653348a694cd1f3ff2496e52, entries=150, sequenceid=766, filesize=11.9 K 2023-04-03 05:56:33,263 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 67f8cb6d380f875bf3bd14331600fad6 in 850ms, sequenceid=766, compaction requested=false 2023-04-03 05:56:33,263 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:33,263 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:33,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:33,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:33,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:33,263 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:33,265 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,265 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,265 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,265 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:33,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:33,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77ed000b[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:33,267 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:33,267 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,268 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,269 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:33,269 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:33,269 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,269 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:33,269 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:33,271 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,271 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,271 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:33,271 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:33,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:33,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:33,273 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:33,273 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:33,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:33,274 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:33,274 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:33,275 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,275 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,275 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:33,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,276 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,277 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,279 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,279 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:33,279 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:33,279 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:33,279 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:33,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:33,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:33,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:33,279 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,279 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,279 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,279 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,280 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,280 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:33,280 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:33,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:33,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:33,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:33,308 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 05:56:33,308 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:33,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:33,308 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:33,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:33,308 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:33,308 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:33,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:33,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501453325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:33,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 431 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501453327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,380 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:33,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:33,382 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:33,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:33,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:33,383 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:33,383 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:33,383 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:33,383 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:33,383 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,383 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,385 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:33,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:33,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,385 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:33,386 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:33,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:33,386 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:33,386 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:33,386 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:33,386 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:33,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,387 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:33,387 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:33,389 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,389 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,390 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,390 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:33,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32ef380a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:33,391 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:33,391 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,392 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,393 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:33,393 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:33,393 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:33,393 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,393 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:33,394 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,394 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:33,394 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:33,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,394 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:33,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:33,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:33,397 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:33,397 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:33,397 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:33,397 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:33,397 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:33,399 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,399 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,399 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:33,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,399 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:33,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,400 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:33,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:33,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:33,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:33,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:33,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:33,403 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:33,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,404 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:33,404 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:33,404 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,404 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:33,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:33,404 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,404 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,404 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:33,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501453428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:33,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 433 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501453430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,504 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:33,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:33,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:33,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:33,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:33,509 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:33,509 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:33,509 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:33,509 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:33,510 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,510 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,511 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:33,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:33,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,511 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:33,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:33,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,512 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:33,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:33,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:33,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:33,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:33,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:33,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:33,513 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:33,513 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:33,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:33,513 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:33,513 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:33,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:33,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:33,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:33,513 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:33,514 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,514 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,515 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,515 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:33,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3890c10d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:33,516 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:33,516 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,517 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,517 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:33,517 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:33,517 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:33,517 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,517 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:33,520 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:33,520 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,520 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:33,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,520 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:33,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:33,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:33,522 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:33,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:33,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:33,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:33,522 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:33,524 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,524 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,524 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:33,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,524 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,526 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,529 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:33,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:33,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:33,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:33,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:33,530 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:33,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,530 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:33,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,530 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:33,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:33,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:33,630 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:33,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:33,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:33,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:33,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:33,633 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:33,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:33,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:33,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:33,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:33,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 452 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501453631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,635 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:33,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 435 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501453634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,636 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:33,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:33,636 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,636 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:33,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:33,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:33,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:33,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:33,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:33,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:33,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:33,637 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:33,638 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:33,638 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:33,638 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:33,638 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:33,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:33,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:33,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:33,638 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:33,639 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,639 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:33,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f7313d3[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:33,641 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:33,641 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,642 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,642 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:33,642 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:33,642 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:33,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,643 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:33,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:33,644 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,644 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:33,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:33,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:33,646 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:33,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:33,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:33,646 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:33,646 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:33,652 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,652 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,652 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:33,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:33,653 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,654 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,656 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,656 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,656 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,656 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,656 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:33,656 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:33,656 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:33,656 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:33,656 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:33,656 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:33,656 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:33,656 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,656 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:33,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:33,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:33,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:33,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:33,718 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403c24a26b5ce734537933690087d827e76 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c24a26b5ce734537933690087d827e76 2023-04-03 05:56:33,722 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=786, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/8d4bea7652944dae824a9aae1cd624dc 2023-04-03 05:56:33,756 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:33,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:33,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:33,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:33,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:33,760 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:33,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:33,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:33,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:33,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,760 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,761 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:33,761 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:33,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:33,762 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,762 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,762 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,763 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:33,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:33,763 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:33,763 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:33,763 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:33,763 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:33,763 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:33,763 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:33,765 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,765 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,766 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,766 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:33,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1081a76e[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:33,769 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:33,769 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,772 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,772 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:33,772 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:33,772 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:33,772 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,772 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:33,773 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,773 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:33,773 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,773 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:33,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:33,776 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:33,776 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:33,776 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:33,776 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:33,776 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:33,777 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,777 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,777 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:33,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,778 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,779 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,781 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,781 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,781 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,781 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:33,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:33,781 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:33,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:33,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:33,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:33,781 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,781 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:33,781 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:33,781 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,781 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:33,782 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,781 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:33,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:33,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:33,882 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:33,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:33,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:33,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:33,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:33,885 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:33,885 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:33,885 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:33,885 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:33,886 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,886 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,888 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:33,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:33,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:33,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:33,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:33,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:33,890 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:33,890 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:33,890 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:33,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,892 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:33,892 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:33,892 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:33,892 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:33,892 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:33,892 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:33,892 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:33,892 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:33,892 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,893 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:33,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,893 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:33,893 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:33,894 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,894 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,894 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,894 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:33,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5be5e21c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:33,897 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:33,897 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:33,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,898 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:33,898 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:33,899 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:33,899 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:33,899 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,900 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:33,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:33,902 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:33,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:33,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:33,902 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:33,902 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:33,904 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,904 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:33,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:33,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,904 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:33,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:33,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:33,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:33,905 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,906 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:33,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:33,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:33,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:33,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:33,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:33,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:33,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:33,909 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:33,908 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:33,909 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:33,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:33,909 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:33,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:33,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:33,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:33,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:33,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:33,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 437 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501453936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:33,938 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:33,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 454 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501453936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,009 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:34,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:34,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:34,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:34,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:34,012 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:34,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:34,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:34,012 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:34,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,013 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,016 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:34,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:34,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:34,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:34,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,018 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:34,018 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:34,018 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:34,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:34,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:34,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:34,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:34,018 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:34,018 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:34,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:34,018 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:34,018 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:34,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:34,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:34,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,018 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:34,019 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:34,021 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,021 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:34,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:34,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,022 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:34,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:34,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:34,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6543d8c4[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:34,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:34,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,024 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,024 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:34,024 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:34,024 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:34,024 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,024 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:34,025 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:34,025 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:34,025 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,025 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:34,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:34,028 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:34,028 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:34,028 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:34,028 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:34,028 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:34,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,029 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:34,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:34,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:34,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,029 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:34,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:34,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,031 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:34,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,034 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,034 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:34,034 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:34,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:34,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:34,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:34,034 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:34,034 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:34,034 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,034 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,034 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,034 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,034 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:34,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:34,034 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:34,035 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:34,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:34,130 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/6d0db8e4013d4e2f93200060aa7e5126 2023-04-03 05:56:34,135 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:34,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:34,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:34,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:34,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:34,151 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:34,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:34,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:34,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:34,151 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,152 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,153 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,153 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:34,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:34,153 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,153 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:34,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:34,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,155 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:34,156 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:34,156 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:34,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:34,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:34,156 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:34,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:34,180 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:34,180 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/3f800a1f00984d1bb4fd9e386201e1b7 2023-04-03 05:56:34,180 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:34,184 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:34,184 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:34,184 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:34,184 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:34,184 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:34,184 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,184 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:34,184 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:34,185 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/8d4bea7652944dae824a9aae1cd624dc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8d4bea7652944dae824a9aae1cd624dc 2023-04-03 05:56:34,186 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,186 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:34,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:34,187 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,187 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:34,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:34,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7239b234[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:34,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:34,189 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8d4bea7652944dae824a9aae1cd624dc, entries=200, sequenceid=786, filesize=32.3 K 2023-04-03 05:56:34,190 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/6d0db8e4013d4e2f93200060aa7e5126 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6d0db8e4013d4e2f93200060aa7e5126 2023-04-03 05:56:34,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,190 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,190 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:34,190 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:34,190 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:34,190 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:34,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:34,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:34,193 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:34,193 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:34,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:34,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:34,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:34,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:34,197 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:34,197 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:34,197 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:34,197 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:34,197 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:34,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:34,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,198 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:34,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:34,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,200 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6d0db8e4013d4e2f93200060aa7e5126, entries=150, sequenceid=786, filesize=11.9 K 2023-04-03 05:56:34,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:34,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/3f800a1f00984d1bb4fd9e386201e1b7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/3f800a1f00984d1bb4fd9e386201e1b7 2023-04-03 05:56:34,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,202 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:34,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,205 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,205 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,205 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,205 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,206 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,206 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,206 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,205 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:34,206 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:34,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:34,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:34,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:34,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:34,206 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:34,206 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:34,206 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:34,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:34,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:34,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/3f800a1f00984d1bb4fd9e386201e1b7, entries=150, sequenceid=786, filesize=11.9 K 2023-04-03 05:56:34,209 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 67f8cb6d380f875bf3bd14331600fad6 in 902ms, sequenceid=786, compaction requested=true 2023-04-03 05:56:34,209 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:34,209 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:34,209 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:34,209 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:34,209 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:34,209 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:34,210 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 86132 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:34,210 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/A is initiating minor compaction (all files) 2023-04-03 05:56:34,210 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/A in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:34,210 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/15fd6f7755894e448849981db4dd97ad, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8893b2510c504aa7ab34914625316caf, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8d4bea7652944dae824a9aae1cd624dc] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=84.1 K 2023-04-03 05:56:34,214 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 15fd6f7755894e448849981db4dd97ad, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680501391527 2023-04-03 05:56:34,214 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:34,214 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/B is initiating minor compaction (all files) 2023-04-03 05:56:34,214 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/B in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:34,214 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/5ffce3f0d7d0473d8b4adf787e6ca699, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/01ce0b6db23c4d8aaf637e95dc59d83b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6d0db8e4013d4e2f93200060aa7e5126] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=36.6 K 2023-04-03 05:56:34,214 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8893b2510c504aa7ab34914625316caf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680501391545 2023-04-03 05:56:34,214 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 5ffce3f0d7d0473d8b4adf787e6ca699, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680501391527 2023-04-03 05:56:34,215 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 8d4bea7652944dae824a9aae1cd624dc, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680501392692 2023-04-03 05:56:34,215 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 01ce0b6db23c4d8aaf637e95dc59d83b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680501391545 2023-04-03 05:56:34,215 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6d0db8e4013d4e2f93200060aa7e5126, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680501392694 2023-04-03 05:56:34,222 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#B#compaction#645 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:34,240 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#A#compaction#644 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:34,247 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/b01ed9a1ce1d4039866996a6c38c779e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b01ed9a1ce1d4039866996a6c38c779e 2023-04-03 05:56:34,256 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/B of 67f8cb6d380f875bf3bd14331600fad6 into b01ed9a1ce1d4039866996a6c38c779e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:34,256 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:34,256 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/B, priority=13, startTime=1680501394209; duration=0sec 2023-04-03 05:56:34,256 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:34,256 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-03 05:56:34,257 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-03 05:56:34,257 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/C is initiating minor compaction (all files) 2023-04-03 05:56:34,257 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/C in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:34,257 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ca7207bc83d04712b6fe17a50abdaa34, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/272772f5653348a694cd1f3ff2496e52, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/3f800a1f00984d1bb4fd9e386201e1b7] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=36.6 K 2023-04-03 05:56:34,258 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting ca7207bc83d04712b6fe17a50abdaa34, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=750, earliestPutTs=1680501391527 2023-04-03 05:56:34,258 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 272772f5653348a694cd1f3ff2496e52, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1680501391545 2023-04-03 05:56:34,259 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 3f800a1f00984d1bb4fd9e386201e1b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680501392694 2023-04-03 05:56:34,282 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#C#compaction#646 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:34,302 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/3803e73d065f44b2b35372c1274dead7 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/3803e73d065f44b2b35372c1274dead7 2023-04-03 05:56:34,304 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/290245ad647340ff8e2a25b30f1950cd as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/290245ad647340ff8e2a25b30f1950cd 2023-04-03 05:56:34,306 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:34,306 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/A of 67f8cb6d380f875bf3bd14331600fad6 into 3803e73d065f44b2b35372c1274dead7(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:34,306 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:34,307 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/A, priority=13, startTime=1680501394209; duration=0sec 2023-04-03 05:56:34,307 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:34,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:34,309 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/C of 67f8cb6d380f875bf3bd14331600fad6 into 290245ad647340ff8e2a25b30f1950cd(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:34,309 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:34,309 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/C, priority=13, startTime=1680501394209; duration=0sec 2023-04-03 05:56:34,309 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:34,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:34,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:34,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:34,310 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:34,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:34,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:34,310 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:34,311 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,311 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,312 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:34,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:34,312 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,313 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,313 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:34,313 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:34,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:34,314 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:34,314 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:34,314 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:34,314 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:34,314 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:34,314 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:34,314 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:34,314 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:34,314 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:34,315 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-03 05:56:34,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:34,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:34,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:34,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:34,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:34,315 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:34,341 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304031a8e0cc98346443cb89d4f22460a8354 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031a8e0cc98346443cb89d4f22460a8354 2023-04-03 05:56:34,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:34,440 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:34,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:34,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 446 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501454456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,458 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:34,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 463 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501454457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:34,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 448 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501454559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:34,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 465 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501454559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,745 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=804, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/6e8271d9bf70478a95c0b95ccd33a443 2023-04-03 05:56:34,756 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=804 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/542928d448344d0fb78bd1454ce9197a 2023-04-03 05:56:34,763 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=804 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/a07eddf9adff487c8a8e946610599275 2023-04-03 05:56:34,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:34,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 450 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501454762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,766 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/6e8271d9bf70478a95c0b95ccd33a443 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/6e8271d9bf70478a95c0b95ccd33a443 2023-04-03 05:56:34,769 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/6e8271d9bf70478a95c0b95ccd33a443, entries=150, sequenceid=804, filesize=25.4 K 2023-04-03 05:56:34,770 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/542928d448344d0fb78bd1454ce9197a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/542928d448344d0fb78bd1454ce9197a 2023-04-03 05:56:34,773 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/542928d448344d0fb78bd1454ce9197a, entries=150, sequenceid=804, filesize=11.9 K 2023-04-03 05:56:34,773 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/a07eddf9adff487c8a8e946610599275 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/a07eddf9adff487c8a8e946610599275 2023-04-03 05:56:34,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:34,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 467 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501454773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,776 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/a07eddf9adff487c8a8e946610599275, entries=150, sequenceid=804, filesize=11.9 K 2023-04-03 05:56:34,777 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 67f8cb6d380f875bf3bd14331600fad6 in 462ms, sequenceid=804, compaction requested=false 2023-04-03 05:56:34,777 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:34,777 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:34,777 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:34,777 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:34,777 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,777 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:34,777 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:34,780 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:34,780 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:34,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:34,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,781 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:34,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:34,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:34,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:34,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f9ec1ca[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:34,782 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:34,782 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,783 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:34,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:34,783 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,784 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:34,783 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:34,785 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,785 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:34,785 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:34,785 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:34,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:34,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:34,788 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:34,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:34,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:34,788 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:34,788 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:34,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:34,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,793 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:34,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:34,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:34,793 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:34,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,797 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:34,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,801 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,801 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,801 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:34,801 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,801 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:34,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:34,801 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:34,801 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,802 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:34,802 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,802 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:34,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:34,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:34,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:34,802 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:34,802 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:34,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:34,902 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:34,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:34,905 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:34,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:34,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:34,907 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:34,907 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:34,907 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:34,907 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:34,908 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,908 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,909 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,909 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:34,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:34,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:34,910 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,910 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:34,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:34,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:34,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:34,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:34,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:34,911 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:34,912 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:34,912 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:34,912 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 05:56:34,912 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:34,912 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:34,912 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:34,912 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:34,912 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:34,912 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:34,933 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040361480bd1b9eb4c84a48d983f95a2a2cd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040361480bd1b9eb4c84a48d983f95a2a2cd 2023-04-03 05:56:34,941 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=821, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/74a41e9ab8604fcb91b7095baf4d4207 2023-04-03 05:56:34,953 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=821 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/9aa38ca72046417788fc6c1bf1d267ae 2023-04-03 05:56:34,961 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=821 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/32f43c42c5fa43ef932450e8b1693cc5 2023-04-03 05:56:34,977 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/74a41e9ab8604fcb91b7095baf4d4207 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/74a41e9ab8604fcb91b7095baf4d4207 2023-04-03 05:56:34,981 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/74a41e9ab8604fcb91b7095baf4d4207, entries=150, sequenceid=821, filesize=25.4 K 2023-04-03 05:56:34,981 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/9aa38ca72046417788fc6c1bf1d267ae as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9aa38ca72046417788fc6c1bf1d267ae 2023-04-03 05:56:34,991 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9aa38ca72046417788fc6c1bf1d267ae, entries=150, sequenceid=821, filesize=11.9 K 2023-04-03 05:56:34,991 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/32f43c42c5fa43ef932450e8b1693cc5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/32f43c42c5fa43ef932450e8b1693cc5 2023-04-03 05:56:34,996 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/32f43c42c5fa43ef932450e8b1693cc5, entries=150, sequenceid=821, filesize=11.9 K 2023-04-03 05:56:34,997 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 67f8cb6d380f875bf3bd14331600fad6 in 85ms, sequenceid=821, compaction requested=true 2023-04-03 05:56:34,997 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:34,997 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:34,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:34,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:34,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:34,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:34,997 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:35,005 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,005 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,005 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,005 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:35,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:35,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b1f2b27[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:35,007 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:35,007 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,009 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:35,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:35,010 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:35,010 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,010 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:35,011 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,011 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:35,011 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:35,011 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:35,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:35,014 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:35,014 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:35,014 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:35,014 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:35,014 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:35,015 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,015 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,015 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,016 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,017 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:35,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:35,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:35,019 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:35,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:35,019 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:35,019 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:35,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:35,019 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:35,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:35,019 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,020 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:35,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:35,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:35,097 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:35,097 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:35,097 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:35,097 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:35,097 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:35,097 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:35,097 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:35,120 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:35,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:35,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:35,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:35,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:35,123 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:35,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:35,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:35,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:35,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,123 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,125 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:35,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:35,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:35,125 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,126 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,126 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:35,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:35,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:35,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:35,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:35,127 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:35,127 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:35,127 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:35,127 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:35,127 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:35,129 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,129 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,130 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:35,131 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60e14578[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:35,131 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:35,131 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,132 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,132 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:35,132 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:35,132 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:35,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,133 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:35,134 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,134 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:35,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,134 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:35,134 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:35,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:35,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:35,137 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:35,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:35,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:35,137 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:35,137 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:35,141 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,141 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,141 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:35,141 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,143 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:35,145 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:35,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:35,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:35,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:35,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,145 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:35,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:35,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:35,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:35,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:35,146 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:35,146 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:35,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501395137, max timestamp=1680501395145Num uniques 859; 2023-04-03 05:56:35,147 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:35,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501395137, max timestamp=1680501395145Num uniques 859; 2023-04-03 05:56:35,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501395137, max timestamp=1680501395145Num uniques 859; 2023-04-03 05:56:35,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:35,148 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:35,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:35,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 472 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501455174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,182 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:35,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 489 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501455178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,246 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:35,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:35,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:35,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:35,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:35,250 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:35,250 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:35,250 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:35,250 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:35,251 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,251 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,252 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:35,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:35,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:35,252 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,252 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:35,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:35,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:35,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:35,254 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:35,254 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:35,254 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:35,254 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:35,254 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:35,254 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:35,256 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,256 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,256 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,256 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:35,258 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f76e342[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:35,258 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:35,258 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,261 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:35,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:35,261 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:35,263 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,263 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:35,266 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,266 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:35,266 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,266 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:35,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:35,269 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:35,269 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:35,269 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:35,269 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:35,269 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:35,270 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,270 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,270 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:35,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,271 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,272 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,277 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:35,278 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:35,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:35,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:35,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:35,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:35,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:35,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,278 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:35,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:35,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:35,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,278 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:35,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,278 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,279 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:35,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 474 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501455279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,286 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:35,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 491 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501455282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,379 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:35,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:35,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:35,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:35,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:35,382 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:35,382 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:35,382 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:35,382 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:35,382 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,382 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,384 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,384 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:35,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:35,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:35,384 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,384 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:35,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:35,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,385 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:35,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:35,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:35,385 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:35,385 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:35,385 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:35,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:35,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:35,386 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:35,386 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:35,386 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:35,386 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:35,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:35,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:35,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:35,386 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:35,387 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,388 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,388 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,388 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:35,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:35,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46fff9f7[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:35,389 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:35,389 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,390 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,390 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,390 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:35,390 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:35,390 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:35,391 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:35,395 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,395 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:35,396 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:35,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:35,397 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:35,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:35,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:35,398 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:35,398 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:35,399 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,399 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,399 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:35,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,399 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,401 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:35,403 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:35,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:35,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:35,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:35,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:35,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:35,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:35,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,403 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:35,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,403 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:35,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:35,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:35,403 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,404 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,486 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:35,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 476 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501455482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,489 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:35,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 493 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501455487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,503 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:35,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:35,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:35,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:35,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:35,507 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:35,507 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:35,507 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:35,507 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:35,508 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,508 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,509 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:35,509 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:35,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:35,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,510 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,510 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:35,510 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:35,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,511 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:35,511 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:35,511 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:35,512 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:35,512 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:35,512 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:35,512 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:35,512 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:35,513 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403017e3f952b394a99af95840581a35659 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403017e3f952b394a99af95840581a35659 2023-04-03 05:56:35,518 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=831, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/0164515fce5b4aa8a95ac39b0aa65df3 2023-04-03 05:56:35,519 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,519 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,520 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,520 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:35,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:35,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24dbb41c[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:35,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:35,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,523 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,523 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:35,523 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:35,523 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:35,523 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,523 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:35,524 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,525 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:35,525 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:35,525 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:35,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:35,527 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:35,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:35,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:35,527 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:35,527 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:35,528 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:35,528 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,529 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:35,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:35,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:35,529 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:35,531 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,537 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,537 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,537 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:35,537 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:35,537 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:35,537 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:35,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:35,537 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:35,537 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,538 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:35,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:35,538 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,538 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:35,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:35,541 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=831 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/fac2313f975b49df83fbc54707fc4b4e 2023-04-03 05:56:35,551 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=831 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/c375aa13d8ac47b484d3a5bcad91ade9 2023-04-03 05:56:35,555 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/0164515fce5b4aa8a95ac39b0aa65df3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/0164515fce5b4aa8a95ac39b0aa65df3 2023-04-03 05:56:35,558 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/0164515fce5b4aa8a95ac39b0aa65df3, entries=350, sequenceid=831, filesize=52.9 K 2023-04-03 05:56:35,558 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/fac2313f975b49df83fbc54707fc4b4e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/fac2313f975b49df83fbc54707fc4b4e 2023-04-03 05:56:35,563 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/fac2313f975b49df83fbc54707fc4b4e, entries=150, sequenceid=831, filesize=11.9 K 2023-04-03 05:56:35,564 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/c375aa13d8ac47b484d3a5bcad91ade9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c375aa13d8ac47b484d3a5bcad91ade9 2023-04-03 05:56:35,567 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c375aa13d8ac47b484d3a5bcad91ade9, entries=150, sequenceid=831, filesize=11.9 K 2023-04-03 05:56:35,568 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 67f8cb6d380f875bf3bd14331600fad6 in 471ms, sequenceid=831, compaction requested=true 2023-04-03 05:56:35,568 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:35,568 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:35,568 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:35,568 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:35,568 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:35,568 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:35,569 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 133408 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:35,569 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/A is initiating minor compaction (all files) 2023-04-03 05:56:35,569 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/A in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:35,570 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/3803e73d065f44b2b35372c1274dead7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/6e8271d9bf70478a95c0b95ccd33a443, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/74a41e9ab8604fcb91b7095baf4d4207, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/0164515fce5b4aa8a95ac39b0aa65df3] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=130.3 K 2023-04-03 05:56:35,570 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 3803e73d065f44b2b35372c1274dead7, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680501392694 2023-04-03 05:56:35,570 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:35,570 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/B is initiating minor compaction (all files) 2023-04-03 05:56:35,570 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/B in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:35,570 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b01ed9a1ce1d4039866996a6c38c779e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/542928d448344d0fb78bd1454ce9197a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9aa38ca72046417788fc6c1bf1d267ae, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/fac2313f975b49df83fbc54707fc4b4e] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=48.6 K 2023-04-03 05:56:35,571 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 6e8271d9bf70478a95c0b95ccd33a443, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1680501393323 2023-04-03 05:56:35,571 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting b01ed9a1ce1d4039866996a6c38c779e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680501392694 2023-04-03 05:56:35,571 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 74a41e9ab8604fcb91b7095baf4d4207, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=821, earliestPutTs=1680501394450 2023-04-03 05:56:35,571 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 542928d448344d0fb78bd1454ce9197a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1680501393323 2023-04-03 05:56:35,571 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 0164515fce5b4aa8a95ac39b0aa65df3, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1680501395080 2023-04-03 05:56:35,572 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 9aa38ca72046417788fc6c1bf1d267ae, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=821, earliestPutTs=1680501394450 2023-04-03 05:56:35,572 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting fac2313f975b49df83fbc54707fc4b4e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1680501395085 2023-04-03 05:56:35,583 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#B#compaction#657 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:35,587 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#A#compaction#656 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:35,604 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/6614470991b8460a8608b5a1c35469db as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6614470991b8460a8608b5a1c35469db 2023-04-03 05:56:35,608 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/B of 67f8cb6d380f875bf3bd14331600fad6 into 6614470991b8460a8608b5a1c35469db(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:35,609 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:35,609 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/B, priority=12, startTime=1680501395568; duration=0sec 2023-04-03 05:56:35,609 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:35,609 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:35,610 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:35,610 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/C is initiating minor compaction (all files) 2023-04-03 05:56:35,610 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/C in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:35,610 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/290245ad647340ff8e2a25b30f1950cd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/a07eddf9adff487c8a8e946610599275, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/32f43c42c5fa43ef932450e8b1693cc5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c375aa13d8ac47b484d3a5bcad91ade9] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=48.6 K 2023-04-03 05:56:35,610 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 290245ad647340ff8e2a25b30f1950cd, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680501392694 2023-04-03 05:56:35,610 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting a07eddf9adff487c8a8e946610599275, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=804, earliestPutTs=1680501393323 2023-04-03 05:56:35,611 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 32f43c42c5fa43ef932450e8b1693cc5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=821, earliestPutTs=1680501394450 2023-04-03 05:56:35,611 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting c375aa13d8ac47b484d3a5bcad91ade9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1680501395085 2023-04-03 05:56:35,626 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#C#compaction#658 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:35,630 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/022bdf3d51f14bd3bcff962db9d576e3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/022bdf3d51f14bd3bcff962db9d576e3 2023-04-03 05:56:35,634 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/A of 67f8cb6d380f875bf3bd14331600fad6 into 022bdf3d51f14bd3bcff962db9d576e3(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:35,634 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:35,634 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/A, priority=12, startTime=1680501395568; duration=0sec 2023-04-03 05:56:35,634 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:35,637 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/34ff5e02af3d47b1bc441c5fad1ba3d3 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/34ff5e02af3d47b1bc441c5fad1ba3d3 2023-04-03 05:56:35,638 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:35,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:35,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:35,641 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/C of 67f8cb6d380f875bf3bd14331600fad6 into 34ff5e02af3d47b1bc441c5fad1ba3d3(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:35,641 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:35,641 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/C, priority=12, startTime=1680501395568; duration=0sec 2023-04-03 05:56:35,641 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:35,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:35,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:35,642 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:35,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:35,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:35,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:35,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,642 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,644 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:35,644 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:35,644 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,644 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:35,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:35,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:35,645 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:35,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:35,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:35,645 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:35,645 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:35,645 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:35,645 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:35,645 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:35,645 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:35,645 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:35,645 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-03 05:56:35,646 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:35,646 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:35,646 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:35,646 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:35,646 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:35,646 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:35,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:35,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:35,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:35,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 482 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501455800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,804 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:35,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 498 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501455801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,905 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:35,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 484 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501455904, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,907 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:35,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 500 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501455906, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:35,977 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-03 05:56:36,056 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040327555e5a5a404a4c84d639dcbefcc0c6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040327555e5a5a404a4c84d639dcbefcc0c6 2023-04-03 05:56:36,060 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=870, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/dfaa672b1bdc492eabd3129545152998 2023-04-03 05:56:36,069 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/1f4935402fc94b939ff6afa2435712f6 2023-04-03 05:56:36,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:36,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 486 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501456107, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:36,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 502 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501456109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:36,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 488 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501456412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:36,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 504 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501456414, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,484 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/a0147f052bd54e94a5e6558dc49f6765 2023-04-03 05:56:36,487 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/dfaa672b1bdc492eabd3129545152998 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/dfaa672b1bdc492eabd3129545152998 2023-04-03 05:56:36,490 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/dfaa672b1bdc492eabd3129545152998, entries=150, sequenceid=870, filesize=25.4 K 2023-04-03 05:56:36,490 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/1f4935402fc94b939ff6afa2435712f6 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1f4935402fc94b939ff6afa2435712f6 2023-04-03 05:56:36,493 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1f4935402fc94b939ff6afa2435712f6, entries=150, sequenceid=870, filesize=11.9 K 2023-04-03 05:56:36,494 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/a0147f052bd54e94a5e6558dc49f6765 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/a0147f052bd54e94a5e6558dc49f6765 2023-04-03 05:56:36,497 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/a0147f052bd54e94a5e6558dc49f6765, entries=150, sequenceid=870, filesize=11.9 K 2023-04-03 05:56:36,498 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 67f8cb6d380f875bf3bd14331600fad6 in 852ms, sequenceid=870, compaction requested=false 2023-04-03 05:56:36,498 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:36,498 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:36,498 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:36,498 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:36,498 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,498 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:36,498 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:36,499 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:36,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:36,499 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:36,500 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,515 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:36,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:36,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:36,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:36,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4efbef7a[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:36,517 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:36,517 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,518 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,518 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:36,518 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:36,518 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:36,518 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,518 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:36,519 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:36,519 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,519 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:36,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:36,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:36,519 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:36,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:36,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:36,522 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:36,522 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:36,522 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:36,523 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:36,523 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:36,523 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:36,526 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,526 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:36,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,526 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:36,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:36,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:36,526 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:36,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,528 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:36,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:36,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:36,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,530 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:36,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:36,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:36,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:36,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,531 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,530 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:36,531 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:36,531 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:36,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:36,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:36,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:36,631 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:36,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:36,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:36,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:36,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:36,635 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:36,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:36,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:36,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:36,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,635 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:36,637 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:36,637 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:36,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:36,637 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,637 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:36,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:36,639 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:36,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:36,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:36,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:36,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:36,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:36,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:36,640 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:36,648 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:36,648 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:36,648 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:36,648 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:36,648 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:36,648 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:36,649 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:36,649 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:36,649 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:36,649 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:36,649 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:36,685 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403b12de7e0e3e1461bb1bfcd037bf5f87c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b12de7e0e3e1461bb1bfcd037bf5f87c 2023-04-03 05:56:36,692 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=880, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/49f0dc3625d3406b9df1e5a744fed6b9 2023-04-03 05:56:36,702 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=880 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/2c4798afb9d5428da40cc8f7bdeba678 2023-04-03 05:56:36,711 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=880 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/98617d8b5baf43b5af95d72ebe87503d 2023-04-03 05:56:36,714 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/49f0dc3625d3406b9df1e5a744fed6b9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/49f0dc3625d3406b9df1e5a744fed6b9 2023-04-03 05:56:36,718 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/49f0dc3625d3406b9df1e5a744fed6b9, entries=150, sequenceid=880, filesize=25.4 K 2023-04-03 05:56:36,718 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/2c4798afb9d5428da40cc8f7bdeba678 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2c4798afb9d5428da40cc8f7bdeba678 2023-04-03 05:56:36,721 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2c4798afb9d5428da40cc8f7bdeba678, entries=150, sequenceid=880, filesize=11.9 K 2023-04-03 05:56:36,722 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/98617d8b5baf43b5af95d72ebe87503d as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/98617d8b5baf43b5af95d72ebe87503d 2023-04-03 05:56:36,725 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/98617d8b5baf43b5af95d72ebe87503d, entries=150, sequenceid=880, filesize=11.9 K 2023-04-03 05:56:36,726 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 67f8cb6d380f875bf3bd14331600fad6 in 78ms, sequenceid=880, compaction requested=true 2023-04-03 05:56:36,726 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:36,726 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:36,726 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:36,726 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:36,726 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,726 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:36,726 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:36,737 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,737 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:36,737 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,737 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:36,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:36,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:36,741 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26ff7914[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:36,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:36,741 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,744 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,744 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:36,744 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:36,744 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:36,745 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:36,745 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:36,745 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:36,748 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,750 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:36,750 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:36,750 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:36,750 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:36,750 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:36,750 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:36,750 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:36,754 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,754 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:36,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,755 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:36,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:36,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:36,762 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:36,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,767 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,774 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:36,774 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:36,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:36,775 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,775 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:36,775 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:36,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:36,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:36,775 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,775 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:36,775 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:36,775 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,775 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,775 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,775 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:36,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:36,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:36,875 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:36,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:36,878 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:36,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:36,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:36,879 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:36,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:36,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:36,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:36,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,879 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:36,880 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:36,880 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:36,881 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,881 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:36,881 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:36,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:36,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:36,882 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:36,882 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:36,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:36,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:36,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:36,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:36,882 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:36,882 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:36,882 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:36,883 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:36,883 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:36,883 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:36,884 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,884 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:36,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:36,885 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,885 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:36,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:36,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:36,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:36,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:36,886 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:36,886 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ff60c7d[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:36,887 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,887 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:36,887 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:36,887 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:36,888 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,888 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:36,890 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,890 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:36,890 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:36,890 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:36,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:36,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:36,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:36,892 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:36,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:36,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:36,892 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:36,892 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:36,894 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,894 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:36,894 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:36,894 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:36,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:36,895 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:36,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:36,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,897 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:36,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:36,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:36,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:36,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:36,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:36,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:36,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:36,898 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:36,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:36,898 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:36,897 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:36,898 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:36,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:36,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:36,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:36,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:36,951 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:36,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:36,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:36,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:36,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:36,952 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:36,952 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:36,981 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403164109c979784941981497255a868114 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403164109c979784941981497255a868114 2023-04-03 05:56:36,998 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:36,998 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=892, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/056463fabd2546669641568e705e1e2e 2023-04-03 05:56:37,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501396987, max timestamp=1680501396997Num uniques 859; 2023-04-03 05:56:37,000 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:37,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:37,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501396987, max timestamp=1680501396999Num uniques 859; 2023-04-03 05:56:37,001 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:37,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680501396987, max timestamp=1680501396997Num uniques 859; 2023-04-03 05:56:37,002 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-03 05:56:37,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:37,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:37,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:37,020 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:37,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:37,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:37,020 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:37,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,021 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,022 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:37,022 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:37,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:37,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,023 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:37,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:37,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:37,036 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:37,036 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:37,036 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:37,036 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:37,036 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:37,046 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 511 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501457041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,049 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 526 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501457046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,066 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:37,067 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:37,067 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/8ce82c75e6074c1c85b063c82eb9d493 2023-04-03 05:56:37,071 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,072 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:37,072 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:37,072 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:37,072 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:37,075 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,075 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,075 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,075 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:37,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66f420cb[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:37,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:37,077 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,078 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:37,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:37,079 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:37,079 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,079 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:37,080 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:37,080 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:37,080 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:37,080 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:37,083 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:37,083 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:37,083 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:37,083 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:37,084 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:37,084 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:37,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,085 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,086 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,087 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,091 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:37,091 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:37,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:37,091 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:37,091 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:37,091 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:37,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:37,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:37,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:37,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:37,091 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,092 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,092 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:37,092 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:37,092 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,092 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,092 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,102 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=892 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/1267d062e44640b0be99a94dc7229af5 2023-04-03 05:56:37,108 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/056463fabd2546669641568e705e1e2e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/056463fabd2546669641568e705e1e2e 2023-04-03 05:56:37,114 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/056463fabd2546669641568e705e1e2e, entries=400, sequenceid=892, filesize=59.8 K 2023-04-03 05:56:37,114 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/8ce82c75e6074c1c85b063c82eb9d493 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8ce82c75e6074c1c85b063c82eb9d493 2023-04-03 05:56:37,119 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8ce82c75e6074c1c85b063c82eb9d493, entries=150, sequenceid=892, filesize=11.9 K 2023-04-03 05:56:37,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/1267d062e44640b0be99a94dc7229af5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/1267d062e44640b0be99a94dc7229af5 2023-04-03 05:56:37,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/1267d062e44640b0be99a94dc7229af5, entries=150, sequenceid=892, filesize=11.9 K 2023-04-03 05:56:37,128 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 67f8cb6d380f875bf3bd14331600fad6 in 177ms, sequenceid=892, compaction requested=true 2023-04-03 05:56:37,128 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:37,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:37,128 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:37,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:37,128 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:37,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:37,129 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 140584 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:37,129 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/A is initiating minor compaction (all files) 2023-04-03 05:56:37,129 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/A in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,129 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/022bdf3d51f14bd3bcff962db9d576e3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/dfaa672b1bdc492eabd3129545152998, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/49f0dc3625d3406b9df1e5a744fed6b9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/056463fabd2546669641568e705e1e2e] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=137.3 K 2023-04-03 05:56:37,130 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 022bdf3d51f14bd3bcff962db9d576e3, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1680501395085 2023-04-03 05:56:37,130 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49936 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:37,131 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/B is initiating minor compaction (all files) 2023-04-03 05:56:37,131 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/B in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,131 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6614470991b8460a8608b5a1c35469db, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1f4935402fc94b939ff6afa2435712f6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2c4798afb9d5428da40cc8f7bdeba678, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8ce82c75e6074c1c85b063c82eb9d493] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=48.8 K 2023-04-03 05:56:37,131 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 6614470991b8460a8608b5a1c35469db, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1680501395085 2023-04-03 05:56:37,131 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting dfaa672b1bdc492eabd3129545152998, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=870, earliestPutTs=1680501395163 2023-04-03 05:56:37,132 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 1f4935402fc94b939ff6afa2435712f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=870, earliestPutTs=1680501395163 2023-04-03 05:56:37,133 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 49f0dc3625d3406b9df1e5a744fed6b9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=880, earliestPutTs=1680501395797 2023-04-03 05:56:37,133 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 2c4798afb9d5428da40cc8f7bdeba678, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=880, earliestPutTs=1680501395797 2023-04-03 05:56:37,133 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 056463fabd2546669641568e705e1e2e, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1680501396929 2023-04-03 05:56:37,134 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] compactions.Compactor(207): Compacting 8ce82c75e6074c1c85b063c82eb9d493, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1680501396947 2023-04-03 05:56:37,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:37,150 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-03 05:56:37,149 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#B#compaction#668 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:37,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:37,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:37,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:37,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:37,150 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:37,150 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:37,173 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#A#compaction#669 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:37,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 531 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501457167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 518 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501457174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,192 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:37,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:37,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:37,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:37,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:37,196 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:37,196 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:37,196 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:37,196 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:37,197 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,197 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,199 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:37,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:37,199 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,199 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,200 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:37,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:37,215 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:37,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:37,215 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:37,215 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:37,215 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:37,215 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:37,216 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:37,216 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:37,216 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:37,216 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:37,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:37,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:37,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:37,218 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,218 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,219 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,219 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:37,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:37,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77e59350[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:37,222 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:37,222 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,223 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,223 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:37,223 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:37,223 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:37,223 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,224 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:37,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:37,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:37,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:37,226 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:37,231 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:37,231 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:37,231 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:37,231 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:37,231 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:37,231 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:37,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:37,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:37,234 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,240 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,246 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/9722027742a24d19a5d2b28f3c74473f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/9722027742a24d19a5d2b28f3c74473f 2023-04-03 05:56:37,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,248 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:37,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:37,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:37,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:37,248 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:37,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:37,249 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:37,249 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:37,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:37,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:37,253 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/A of 67f8cb6d380f875bf3bd14331600fad6 into 9722027742a24d19a5d2b28f3c74473f(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:37,253 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:37,253 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/A, priority=12, startTime=1680501397128; duration=0sec 2023-04-03 05:56:37,254 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:37,254 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-03 05:56:37,255 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49936 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-03 05:56:37,255 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1912): 67f8cb6d380f875bf3bd14331600fad6/C is initiating minor compaction (all files) 2023-04-03 05:56:37,255 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 67f8cb6d380f875bf3bd14331600fad6/C in TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,255 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/34ff5e02af3d47b1bc441c5fad1ba3d3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/a0147f052bd54e94a5e6558dc49f6765, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/98617d8b5baf43b5af95d72ebe87503d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/1267d062e44640b0be99a94dc7229af5] into tmpdir=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp, totalSize=48.8 K 2023-04-03 05:56:37,256 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 34ff5e02af3d47b1bc441c5fad1ba3d3, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1680501395085 2023-04-03 05:56:37,256 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting a0147f052bd54e94a5e6558dc49f6765, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=870, earliestPutTs=1680501395163 2023-04-03 05:56:37,256 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 98617d8b5baf43b5af95d72ebe87503d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=880, earliestPutTs=1680501395797 2023-04-03 05:56:37,257 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] compactions.Compactor(207): Compacting 1267d062e44640b0be99a94dc7229af5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=892, earliestPutTs=1680501396947 2023-04-03 05:56:37,265 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] throttle.PressureAwareThroughputController(145): 67f8cb6d380f875bf3bd14331600fad6#C#compaction#671 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-03 05:56:37,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 533 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501457275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,278 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/05dc68d8ad6f45af8e1c166b9ea56212 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/05dc68d8ad6f45af8e1c166b9ea56212 2023-04-03 05:56:37,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 520 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501457279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,282 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/C of 67f8cb6d380f875bf3bd14331600fad6 into 05dc68d8ad6f45af8e1c166b9ea56212(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:37,283 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:37,283 INFO [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/C, priority=12, startTime=1680501397128; duration=0sec 2023-04-03 05:56:37,283 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:37,349 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:37,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:37,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:37,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:37,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:37,352 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:37,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:37,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:37,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:37,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,352 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,355 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:37,355 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:37,355 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,355 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:37,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:37,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:37,356 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:37,356 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:37,356 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,356 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:37,356 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:37,357 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:37,357 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:37,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:37,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:37,357 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:37,358 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,358 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,358 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,359 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:37,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a0f6361[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:37,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:37,360 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,361 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,361 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:37,361 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:37,361 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:37,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,361 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:37,362 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,362 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:37,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:37,363 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:37,365 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:37,365 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:37,365 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:37,365 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:37,365 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:37,365 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:37,368 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,368 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,368 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:37,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:37,369 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,370 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,372 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:37,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:37,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:37,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:37,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:37,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:37,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:37,372 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:37,372 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:37,373 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:37,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:37,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:37,473 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:37,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:37,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:37,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:37,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:37,478 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:37,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:37,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:37,478 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:37,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,479 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,480 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:37,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:37,480 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,481 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:37,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] ipc.CallRunner(144): callId: 535 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501457479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:37,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:37,482 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:37,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:37,482 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:37,482 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:37,482 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:37,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 522 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501457482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:37,485 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,485 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:37,485 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:37,485 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:37,485 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:37,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:37,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:37,486 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:37,487 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,487 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,487 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,487 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:37,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:37,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53517b9[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:37,489 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:37,489 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,492 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,492 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:37,492 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:37,492 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:37,492 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,492 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:37,493 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,494 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:37,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,494 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:37,494 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:37,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:37,497 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:37,497 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:37,497 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:37,497 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:37,497 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:37,499 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,499 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:37,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,499 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,500 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,502 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:37,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:37,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:37,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:37,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:37,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,505 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:37,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:37,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:37,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,505 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:37,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:37,506 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:37,506 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:37,577 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/5ba1a7df2e05474ea43e5871ae03cebf as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/5ba1a7df2e05474ea43e5871ae03cebf 2023-04-03 05:56:37,580 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 67f8cb6d380f875bf3bd14331600fad6/B of 67f8cb6d380f875bf3bd14331600fad6 into 5ba1a7df2e05474ea43e5871ae03cebf(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-03 05:56:37,580 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:37,580 INFO [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., storeName=67f8cb6d380f875bf3bd14331600fad6/B, priority=12, startTime=1680501397128; duration=0sec 2023-04-03 05:56:37,580 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:37,599 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304032f897130a385470e9878da4f7cad49a8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032f897130a385470e9878da4f7cad49a8 2023-04-03 05:56:37,603 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=929, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/d68e2c1852b247a6a1c268669d084edc 2023-04-03 05:56:37,606 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:37,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:37,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:37,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:37,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:37,610 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:37,610 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:37,610 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:37,610 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:37,611 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,611 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,612 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,613 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:37,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:37,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:37,614 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,614 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:37,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,614 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=929 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/e2e86a01e059438ba719772bcc85e314 2023-04-03 05:56:37,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:37,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:37,615 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:37,615 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:37,615 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:37,615 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:37,615 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:37,616 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:37,616 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:37,616 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:37,616 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:37,616 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:37,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,618 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:37,618 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:37,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42008dac[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:37,625 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:37,625 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,626 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:37,627 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,627 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:37,627 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:37,627 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:37,629 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:37,629 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:37,629 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:37,629 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:37,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:37,633 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:37,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:37,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:37,633 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:37,633 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:37,635 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=929 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/7cb37ab84dce4b1aae8e965d837ddab5 2023-04-03 05:56:37,637 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,637 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:37,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,637 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:37,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:37,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:37,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:37,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:37,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,638 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:37,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,640 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:37,641 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/d68e2c1852b247a6a1c268669d084edc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d68e2c1852b247a6a1c268669d084edc 2023-04-03 05:56:37,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:37,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,643 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:37,643 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:37,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:37,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:37,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:37,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:37,643 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:37,643 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:37,643 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,644 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,644 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,644 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,644 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:37,644 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,643 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:37,643 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:37,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:37,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:37,647 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d68e2c1852b247a6a1c268669d084edc, entries=200, sequenceid=929, filesize=32.3 K 2023-04-03 05:56:37,648 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/e2e86a01e059438ba719772bcc85e314 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/e2e86a01e059438ba719772bcc85e314 2023-04-03 05:56:37,652 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/e2e86a01e059438ba719772bcc85e314, entries=150, sequenceid=929, filesize=11.9 K 2023-04-03 05:56:37,652 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/7cb37ab84dce4b1aae8e965d837ddab5 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7cb37ab84dce4b1aae8e965d837ddab5 2023-04-03 05:56:37,657 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7cb37ab84dce4b1aae8e965d837ddab5, entries=150, sequenceid=929, filesize=11.9 K 2023-04-03 05:56:37,658 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 67f8cb6d380f875bf3bd14331600fad6 in 508ms, sequenceid=929, compaction requested=false 2023-04-03 05:56:37,658 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:37,744 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:37,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:37,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:37,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:37,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:37,747 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:37,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:37,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:37,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:37,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,747 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,754 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,754 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:37,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:37,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:37,754 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,754 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:37,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:37,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:37,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:37,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:37,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:37,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:37,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:37,755 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:37,756 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:37,756 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:37,756 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:37,756 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:37,756 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:37,756 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:37,756 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:37,756 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:37,756 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:37,756 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:37,756 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:37,766 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403d6cd3c745d54451cbf363b81a761d737 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d6cd3c745d54451cbf363b81a761d737 2023-04-03 05:56:37,771 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=942, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/47f3145c4f7c4761a9a12d4a9530f236 2023-04-03 05:56:37,778 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=942 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/0772805e9ac14feeabb3418c11bb654f 2023-04-03 05:56:37,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:37,787 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:37,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501397810, max timestamp=1680501397815Num uniques 859; 2023-04-03 05:56:37,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:37,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501397810, max timestamp=1680501397815Num uniques 859; 2023-04-03 05:56:37,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680501397810, max timestamp=1680501397815Num uniques 859; 2023-04-03 05:56:37,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:37,817 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-03 05:56:37,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 540 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501457836, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 554 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501457837, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39448 deadline: 1680501457898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,900 DEBUG [Thread-4121] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:56:37,940 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 542 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501457939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,941 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 556 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501457940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,953 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39462 deadline: 1680501457953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,955 DEBUG [Thread-4113] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18245 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:56:37,971 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:37,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39388 deadline: 1680501457969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:37,973 DEBUG [Thread-4119] client.RpcRetryingCallerImpl(129): Call exception, tries=8, retries=16, started=18279 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6., hostname=jenkins-hbase4.apache.org,37397,1680501214665, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor31.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-03 05:56:38,146 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:38,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 544 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501458143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:38,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 558 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501458143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,187 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=942 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/7e75c7c0a68544cc87d41680d3186558 2023-04-03 05:56:38,191 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/47f3145c4f7c4761a9a12d4a9530f236 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/47f3145c4f7c4761a9a12d4a9530f236 2023-04-03 05:56:38,197 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/47f3145c4f7c4761a9a12d4a9530f236, entries=100, sequenceid=942, filesize=18.6 K 2023-04-03 05:56:38,197 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/0772805e9ac14feeabb3418c11bb654f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/0772805e9ac14feeabb3418c11bb654f 2023-04-03 05:56:38,201 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/0772805e9ac14feeabb3418c11bb654f, entries=100, sequenceid=942, filesize=9.5 K 2023-04-03 05:56:38,202 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/7e75c7c0a68544cc87d41680d3186558 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7e75c7c0a68544cc87d41680d3186558 2023-04-03 05:56:38,205 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7e75c7c0a68544cc87d41680d3186558, entries=100, sequenceid=942, filesize=9.5 K 2023-04-03 05:56:38,206 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 67f8cb6d380f875bf3bd14331600fad6 in 450ms, sequenceid=942, compaction requested=true 2023-04-03 05:56:38,206 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:38,206 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:38,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:38,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:38,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:38,206 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:38,208 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:38,208 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:38,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:38,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:38,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:38,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19a4e9af[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:38,211 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:38,211 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:38,213 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:38,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:38,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:38,213 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:38,213 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:38,213 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:38,213 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,213 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:38,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:38,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:38,216 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:38,216 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:38,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:38,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:38,218 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:38,218 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:38,218 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:38,218 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:38,218 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:38,218 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:38,220 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:38,220 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:38,220 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:38,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:38,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:38,220 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:38,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:38,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:38,222 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:38,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,224 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,224 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:38,224 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:38,224 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:38,224 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:38,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:38,224 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:38,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:38,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:38,224 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:38,224 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:38,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:38,225 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:38,225 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:38,224 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:38,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:38,225 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:38,325 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:38,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:38,326 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:38,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:38,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:38,327 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:38,328 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:38,328 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:38,328 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:38,328 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:38,328 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:38,329 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:38,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:38,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:38,329 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:38,330 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,330 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:38,330 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:38,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:38,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:38,330 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:38,331 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:38,331 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:38,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:38,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:38,331 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:38,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:38,340 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:38,340 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:38,340 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:38,340 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-04-03 05:56:38,340 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:38,340 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:38,340 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:38,340 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:38,341 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:38,341 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-03 05:56:38,376 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403c0abbf39fe5b4521b37c5e5b608e303c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c0abbf39fe5b4521b37c5e5b608e303c 2023-04-03 05:56:38,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:38,454 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:38,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:38,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 549 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501458461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,466 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:38,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 564 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501458464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,566 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:38,566 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 551 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501458565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,569 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:38,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 566 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501458567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,770 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:38,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 553 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501458768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,773 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:38,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 568 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501458771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:38,781 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=978, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/c805f326d74e48c4b34b2b24b7a89e09 2023-04-03 05:56:39,075 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:39,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=37397] ipc.CallRunner(144): callId: 555 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39362 deadline: 1680501459073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-03 05:56:39,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] ipc.CallRunner(144): callId: 570 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39412 deadline: 1680501459075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,194 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=978 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/3bbc71e5ac954df4bf292cae298ef2ca 2023-04-03 05:56:39,204 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=978 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/ddfab1c72afd431f88b462af9ca45b24 2023-04-03 05:56:39,207 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/c805f326d74e48c4b34b2b24b7a89e09 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/c805f326d74e48c4b34b2b24b7a89e09 2023-04-03 05:56:39,209 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/c805f326d74e48c4b34b2b24b7a89e09, entries=150, sequenceid=978, filesize=25.4 K 2023-04-03 05:56:39,210 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/3bbc71e5ac954df4bf292cae298ef2ca as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/3bbc71e5ac954df4bf292cae298ef2ca 2023-04-03 05:56:39,213 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/3bbc71e5ac954df4bf292cae298ef2ca, entries=150, sequenceid=978, filesize=11.9 K 2023-04-03 05:56:39,214 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/ddfab1c72afd431f88b462af9ca45b24 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ddfab1c72afd431f88b462af9ca45b24 2023-04-03 05:56:39,216 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ddfab1c72afd431f88b462af9ca45b24, entries=150, sequenceid=978, filesize=11.9 K 2023-04-03 05:56:39,217 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 67f8cb6d380f875bf3bd14331600fad6 in 877ms, sequenceid=978, compaction requested=true 2023-04-03 05:56:39,217 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:39,217 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:39,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:39,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:39,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:39,217 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:39,219 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,219 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:39,219 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:39,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:39,219 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:39,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:39,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:39,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:39,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:39,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48e18591[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:39,221 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:39,221 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:39,222 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:39,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:39,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:39,222 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:39,222 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:39,222 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:39,222 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,223 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:39,224 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,224 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:39,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:39,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:39,225 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:39,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:39,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:39,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:39,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:39,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:39,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:39,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:39,227 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:39,227 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:39,227 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:39,227 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:39,227 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:39,229 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:39,229 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:39,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:39,229 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:39,229 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:39,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:39,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:39,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:39,230 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:39,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,232 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:39,232 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:39,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:39,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:39,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:39,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:39,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:39,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:39,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:39,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:39,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:39,233 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:39,233 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:39,233 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:39,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:39,233 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:39,233 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:39,333 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:39,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:39,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-03 05:56:39,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-03 05:56:39,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-03 05:56:39,338 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-03 05:56:39,338 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:39,338 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-03 05:56:39,338 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-03 05:56:39,340 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:39,340 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:39,341 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:39,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:39,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:39,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:39,341 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:39,342 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:39,342 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-03 05:56:39,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-03 05:56:39,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:39,342 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-03 05:56:39,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-03 05:56:39,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-03 05:56:39,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-03 05:56:39,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-03 05:56:39,342 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-03 05:56:39,343 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-03 05:56:39,343 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-03 05:56:39,343 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:39,343 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. started... 2023-04-03 05:56:39,343 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-03 05:56:39,344 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:39,344 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:39,344 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:39,344 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:39,344 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:39,344 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:39,373 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403260e6ed9f93448cd9a469045b2c087d0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403260e6ed9f93448cd9a469045b2c087d0 2023-04-03 05:56:39,386 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=988, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/424ded1ba3e043658f0377f753f3842e 2023-04-03 05:56:39,583 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. as already flushing 2023-04-03 05:56:39,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:39,625 DEBUG [Thread-4128] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3420e54c to 127.0.0.1:63744 2023-04-03 05:56:39,625 DEBUG [Thread-4128] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:39,626 DEBUG [Thread-4126] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x17cf2e3f to 127.0.0.1:63744 2023-04-03 05:56:39,626 DEBUG [Thread-4126] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:39,631 DEBUG [Thread-4132] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x601410a4 to 127.0.0.1:63744 2023-04-03 05:56:39,632 DEBUG [Thread-4132] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:39,635 DEBUG [Thread-4130] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x26d7d0b5 to 127.0.0.1:63744 2023-04-03 05:56:39,636 DEBUG [Thread-4130] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:39,636 DEBUG [Thread-4115] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x089b6154 to 127.0.0.1:63744 2023-04-03 05:56:39,636 DEBUG [Thread-4117] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x01d207bb to 127.0.0.1:63744 2023-04-03 05:56:39,636 DEBUG [Thread-4115] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:39,636 DEBUG [Thread-4117] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:39,640 DEBUG [Thread-4124] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x526f2221 to 127.0.0.1:63744 2023-04-03 05:56:39,640 DEBUG [Thread-4124] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:39,809 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=988 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/b3d25e91d0f8465fbd51dec5603f2730 2023-04-03 05:56:40,222 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=988 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/7f23e030c3d94e6c8f536925235e18ac 2023-04-03 05:56:40,226 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/424ded1ba3e043658f0377f753f3842e as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/424ded1ba3e043658f0377f753f3842e 2023-04-03 05:56:40,229 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/424ded1ba3e043658f0377f753f3842e, entries=150, sequenceid=988, filesize=25.4 K 2023-04-03 05:56:40,230 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/b3d25e91d0f8465fbd51dec5603f2730 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b3d25e91d0f8465fbd51dec5603f2730 2023-04-03 05:56:40,233 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b3d25e91d0f8465fbd51dec5603f2730, entries=150, sequenceid=988, filesize=11.9 K 2023-04-03 05:56:40,234 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/7f23e030c3d94e6c8f536925235e18ac as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7f23e030c3d94e6c8f536925235e18ac 2023-04-03 05:56:40,238 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7f23e030c3d94e6c8f536925235e18ac, entries=150, sequenceid=988, filesize=11.9 K 2023-04-03 05:56:40,238 INFO [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=87.22 KB/89310 for 67f8cb6d380f875bf3bd14331600fad6 in 895ms, sequenceid=988, compaction requested=true 2023-04-03 05:56:40,239 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:40,239 DEBUG [rs(jenkins-hbase4.apache.org,37397,1680501214665)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:40,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-03 05:56:40,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-03 05:56:40,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-03 05:56:40,239 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-03 05:56:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:40,242 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-03 05:56:40,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:40,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:40,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:40,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,37397,1680501214665' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-03 05:56:40,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6544cbeb[Count = 0] remaining members to acquire global barrier 2023-04-03 05:56:40,245 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-03 05:56:40,245 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:40,246 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:40,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:40,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:40,246 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-03 05:56:40,246 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-03 05:56:40,246 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,37397,1680501214665' in zk 2023-04-03 05:56:40,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,247 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-03 05:56:40,247 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-03 05:56:40,247 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,248 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:40,248 DEBUG [member: 'jenkins-hbase4.apache.org,37397,1680501214665' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-03 05:56:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:40,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:40,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,37397,1680501214665': 2023-04-03 05:56:40,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,37397,1680501214665' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-03 05:56:40,250 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-03 05:56:40,251 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-03 05:56:40,251 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-03 05:56:40,251 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-03 05:56:40,251 INFO [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-03 05:56:40,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:40,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:40,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-03 05:56:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-03 05:56:40,252 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:40,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-03 05:56:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:40,253 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-03 05:56:40,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:40,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,255 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-03 05:56:40,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-03 05:56:40,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,262 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,262 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:40,262 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-03 05:56:40,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-03 05:56:40,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-03 05:56:40,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-03 05:56:40,262 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-03 05:56:40,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-03 05:56:40,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-03 05:56:40,262 DEBUG [(jenkins-hbase4.apache.org,33301,1680501213465)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-03 05:56:40,262 DEBUG [Thread-4123] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-03 05:56:40,263 DEBUG [Thread-4123] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-03 05:56:40,262 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-03 05:56:40,262 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:40,263 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:40,263 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-03 05:56:40,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-03 05:56:40,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-03 05:56:40,263 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-03 05:56:40,363 DEBUG [Thread-4123] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-03 05:56:40,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-03 05:56:47,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=37397] regionserver.HRegion(9158): Flush requested on 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:47,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-03 05:56:47,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:47,907 DEBUG [Thread-4121] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23afc0f2 to 127.0.0.1:63744 2023-04-03 05:56:47,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:47,907 DEBUG [Thread-4121] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:47,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:47,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:47,907 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:47,907 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:47,915 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403320dc9e29e0d436bbaaf0211c7a741cf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403320dc9e29e0d436bbaaf0211c7a741cf 2023-04-03 05:56:47,918 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1005, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/46de4850990b4e1d84b71c4d2d459d9a 2023-04-03 05:56:47,924 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1005 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/ecf022ac75b54a2ba46583d3fbc968fb 2023-04-03 05:56:47,989 DEBUG [Thread-4113] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x041dc777 to 127.0.0.1:63744 2023-04-03 05:56:47,989 DEBUG [Thread-4113] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:48,058 DEBUG [Thread-4119] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5bcc330a to 127.0.0.1:63744 2023-04-03 05:56:48,058 DEBUG [Thread-4119] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 10 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 409 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 425 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 12 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(394): wrote 11 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 4591 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 4529 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 4696 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 4577 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(398): read 4551 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-03 05:56:48,059 INFO [Listener at localhost/37275] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 05:56:48,059 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x54cb8b37 to 127.0.0.1:63744 2023-04-03 05:56:48,059 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:48,060 INFO [Listener at localhost/37275] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-03 05:56:48,060 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-03 05:56:48,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:48,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 05:56:48,063 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501408063"}]},"ts":"1680501408063"} 2023-04-03 05:56:48,064 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-03 05:56:48,065 INFO [PEWorker-1] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-03 05:56:48,066 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=67f8cb6d380f875bf3bd14331600fad6, UNASSIGN}] 2023-04-03 05:56:48,067 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=67f8cb6d380f875bf3bd14331600fad6, UNASSIGN 2023-04-03 05:56:48,067 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=67f8cb6d380f875bf3bd14331600fad6, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:48,067 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501408067"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680501408067"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680501408067"}]},"ts":"1680501408067"} 2023-04-03 05:56:48,068 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665}] 2023-04-03 05:56:48,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 05:56:48,219 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:48,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 67f8cb6d380f875bf3bd14331600fad6, disabling compactions & flushes 2023-04-03 05:56:48,220 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:48,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=1005 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/b96b57a35bd04b9f8b770aee69225824 2023-04-03 05:56:48,333 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/46de4850990b4e1d84b71c4d2d459d9a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/46de4850990b4e1d84b71c4d2d459d9a 2023-04-03 05:56:48,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/46de4850990b4e1d84b71c4d2d459d9a, entries=150, sequenceid=1005, filesize=25.4 K 2023-04-03 05:56:48,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/ecf022ac75b54a2ba46583d3fbc968fb as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ecf022ac75b54a2ba46583d3fbc968fb 2023-04-03 05:56:48,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ecf022ac75b54a2ba46583d3fbc968fb, entries=150, sequenceid=1005, filesize=11.9 K 2023-04-03 05:56:48,339 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/b96b57a35bd04b9f8b770aee69225824 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b96b57a35bd04b9f8b770aee69225824 2023-04-03 05:56:48,342 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b96b57a35bd04b9f8b770aee69225824, entries=150, sequenceid=1005, filesize=11.9 K 2023-04-03 05:56:48,342 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=13.42 KB/13740 for 67f8cb6d380f875bf3bd14331600fad6 in 436ms, sequenceid=1005, compaction requested=true 2023-04-03 05:56:48,342 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:48,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:48,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:48,343 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. because compaction request was cancelled 2023-04-03 05:56:48,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:48,343 DEBUG [RS:0;jenkins-hbase4:37397-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. because compaction request was cancelled 2023-04-03 05:56:48,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-03 05:56:48,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. after waiting 0 ms 2023-04-03 05:56:48,343 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-03 05:56:48,343 DEBUG [RS:0;jenkins-hbase4:37397-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. because compaction request was cancelled 2023-04-03 05:56:48,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:48,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 67f8cb6d380f875bf3bd14331600fad6 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-03 05:56:48,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=A 2023-04-03 05:56:48,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:48,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=B 2023-04-03 05:56:48,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:48,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 67f8cb6d380f875bf3bd14331600fad6, store=C 2023-04-03 05:56:48,343 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-03 05:56:48,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 05:56:48,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 05:56:48,751 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230403937fc0f37f9848e2a03128f1f062383b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403937fc0f37f9848e2a03128f1f062383b 2023-04-03 05:56:49,155 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1010, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/f6c5ee45d9d243858625fb867f7ed6cc 2023-04-03 05:56:49,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 05:56:49,562 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=1010 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/0ba69ac729334d008ba2995533552406 2023-04-03 05:56:49,971 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=1010 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/94893bf8809b4b3799de642f5090b2b9 2023-04-03 05:56:49,974 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/A/f6c5ee45d9d243858625fb867f7ed6cc as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/f6c5ee45d9d243858625fb867f7ed6cc 2023-04-03 05:56:49,976 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/f6c5ee45d9d243858625fb867f7ed6cc, entries=100, sequenceid=1010, filesize=18.6 K 2023-04-03 05:56:49,977 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/B/0ba69ac729334d008ba2995533552406 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/0ba69ac729334d008ba2995533552406 2023-04-03 05:56:49,980 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/0ba69ac729334d008ba2995533552406, entries=100, sequenceid=1010, filesize=9.5 K 2023-04-03 05:56:49,980 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/.tmp/C/94893bf8809b4b3799de642f5090b2b9 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/94893bf8809b4b3799de642f5090b2b9 2023-04-03 05:56:49,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/94893bf8809b4b3799de642f5090b2b9, entries=100, sequenceid=1010, filesize=9.5 K 2023-04-03 05:56:49,983 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=0 B/0 for 67f8cb6d380f875bf3bd14331600fad6 in 1640ms, sequenceid=1010, compaction requested=true 2023-04-03 05:56:49,984 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/b12c3337eeb14917a787387c4cddb0b7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/963f161354934dce83697a7d2b69bf8c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/be8885a52d704d8b850ec9fb092d3de6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/febc4f8da53b41f1aee7c5b2e949c38d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/0de10612b7324fd28a7edeb3f0e45317, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/06b86f42aec6439abe2cb93b497b4993, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d45d11adef8849ebae46c3de1159542a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8f21d2760dcd46c4b8cdecc93087cd99, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/c72f11e2d2f143749dd1c96699734af9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e5084b3efc90467398833aa9dfeee814, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8e6fdce74d1e472ca2f41995f75172eb, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/02140f2988734197a3d5cc82d0132d88, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e1295a826a5a47f5adcd290b23c73603, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/2b809f486b9b4529b56c1b29ebd0ccad, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/68a3b73ba43e4b69b1edb27f7f3a1a4c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/1fa961ee8be844078903a81a9cd40ae1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d922b092f27249caada936560e5c8738, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/4834ab371fe9445eaaa566ae30bd521a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/38347733a8004ec182d6d97cbb438dbc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/417384cfc3504622b9e6a6c6f539fbc3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/a6174f57cfe843438bb354ffc0465e88, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d22f5287bcb4478681e307372cd8f9f0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8e5177a6fcd241b180a1c59ca7731d3e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/3b272021ed224360892ff45cff7a6789, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e6d00740bd9a451495f73234a1ec1219, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/ab23093bec2f4166b571837c9a5d8e92, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/f1d63db8586746bd8d2b4792cebf8c0b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/07f0c76dd30249898ca4e5089ecab4b4, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/15fd6f7755894e448849981db4dd97ad, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8893b2510c504aa7ab34914625316caf, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8d4bea7652944dae824a9aae1cd624dc, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/3803e73d065f44b2b35372c1274dead7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/6e8271d9bf70478a95c0b95ccd33a443, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/74a41e9ab8604fcb91b7095baf4d4207, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/0164515fce5b4aa8a95ac39b0aa65df3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/022bdf3d51f14bd3bcff962db9d576e3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/dfaa672b1bdc492eabd3129545152998, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/49f0dc3625d3406b9df1e5a744fed6b9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/056463fabd2546669641568e705e1e2e] to archive 2023-04-03 05:56:49,985 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:56:49,986 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/b12c3337eeb14917a787387c4cddb0b7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/b12c3337eeb14917a787387c4cddb0b7 2023-04-03 05:56:49,987 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/963f161354934dce83697a7d2b69bf8c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/963f161354934dce83697a7d2b69bf8c 2023-04-03 05:56:49,988 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/be8885a52d704d8b850ec9fb092d3de6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/be8885a52d704d8b850ec9fb092d3de6 2023-04-03 05:56:49,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/febc4f8da53b41f1aee7c5b2e949c38d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/febc4f8da53b41f1aee7c5b2e949c38d 2023-04-03 05:56:49,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/0de10612b7324fd28a7edeb3f0e45317 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/0de10612b7324fd28a7edeb3f0e45317 2023-04-03 05:56:49,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/06b86f42aec6439abe2cb93b497b4993 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/06b86f42aec6439abe2cb93b497b4993 2023-04-03 05:56:49,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d45d11adef8849ebae46c3de1159542a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d45d11adef8849ebae46c3de1159542a 2023-04-03 05:56:49,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8f21d2760dcd46c4b8cdecc93087cd99 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8f21d2760dcd46c4b8cdecc93087cd99 2023-04-03 05:56:49,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/c72f11e2d2f143749dd1c96699734af9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/c72f11e2d2f143749dd1c96699734af9 2023-04-03 05:56:49,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e5084b3efc90467398833aa9dfeee814 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e5084b3efc90467398833aa9dfeee814 2023-04-03 05:56:49,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8e6fdce74d1e472ca2f41995f75172eb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8e6fdce74d1e472ca2f41995f75172eb 2023-04-03 05:56:49,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/02140f2988734197a3d5cc82d0132d88 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/02140f2988734197a3d5cc82d0132d88 2023-04-03 05:56:49,996 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e1295a826a5a47f5adcd290b23c73603 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e1295a826a5a47f5adcd290b23c73603 2023-04-03 05:56:49,997 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/2b809f486b9b4529b56c1b29ebd0ccad to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/2b809f486b9b4529b56c1b29ebd0ccad 2023-04-03 05:56:49,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/68a3b73ba43e4b69b1edb27f7f3a1a4c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/68a3b73ba43e4b69b1edb27f7f3a1a4c 2023-04-03 05:56:49,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/1fa961ee8be844078903a81a9cd40ae1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/1fa961ee8be844078903a81a9cd40ae1 2023-04-03 05:56:50,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d922b092f27249caada936560e5c8738 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d922b092f27249caada936560e5c8738 2023-04-03 05:56:50,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/4834ab371fe9445eaaa566ae30bd521a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/4834ab371fe9445eaaa566ae30bd521a 2023-04-03 05:56:50,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/38347733a8004ec182d6d97cbb438dbc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/38347733a8004ec182d6d97cbb438dbc 2023-04-03 05:56:50,002 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/417384cfc3504622b9e6a6c6f539fbc3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/417384cfc3504622b9e6a6c6f539fbc3 2023-04-03 05:56:50,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/a6174f57cfe843438bb354ffc0465e88 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/a6174f57cfe843438bb354ffc0465e88 2023-04-03 05:56:50,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d22f5287bcb4478681e307372cd8f9f0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d22f5287bcb4478681e307372cd8f9f0 2023-04-03 05:56:50,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8e5177a6fcd241b180a1c59ca7731d3e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8e5177a6fcd241b180a1c59ca7731d3e 2023-04-03 05:56:50,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/3b272021ed224360892ff45cff7a6789 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/3b272021ed224360892ff45cff7a6789 2023-04-03 05:56:50,006 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e6d00740bd9a451495f73234a1ec1219 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/e6d00740bd9a451495f73234a1ec1219 2023-04-03 05:56:50,007 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/ab23093bec2f4166b571837c9a5d8e92 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/ab23093bec2f4166b571837c9a5d8e92 2023-04-03 05:56:50,008 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/f1d63db8586746bd8d2b4792cebf8c0b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/f1d63db8586746bd8d2b4792cebf8c0b 2023-04-03 05:56:50,009 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/07f0c76dd30249898ca4e5089ecab4b4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/07f0c76dd30249898ca4e5089ecab4b4 2023-04-03 05:56:50,010 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/15fd6f7755894e448849981db4dd97ad to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/15fd6f7755894e448849981db4dd97ad 2023-04-03 05:56:50,011 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8893b2510c504aa7ab34914625316caf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8893b2510c504aa7ab34914625316caf 2023-04-03 05:56:50,012 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8d4bea7652944dae824a9aae1cd624dc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/8d4bea7652944dae824a9aae1cd624dc 2023-04-03 05:56:50,013 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/3803e73d065f44b2b35372c1274dead7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/3803e73d065f44b2b35372c1274dead7 2023-04-03 05:56:50,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/6e8271d9bf70478a95c0b95ccd33a443 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/6e8271d9bf70478a95c0b95ccd33a443 2023-04-03 05:56:50,014 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/74a41e9ab8604fcb91b7095baf4d4207 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/74a41e9ab8604fcb91b7095baf4d4207 2023-04-03 05:56:50,015 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/0164515fce5b4aa8a95ac39b0aa65df3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/0164515fce5b4aa8a95ac39b0aa65df3 2023-04-03 05:56:50,016 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/022bdf3d51f14bd3bcff962db9d576e3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/022bdf3d51f14bd3bcff962db9d576e3 2023-04-03 05:56:50,017 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/dfaa672b1bdc492eabd3129545152998 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/dfaa672b1bdc492eabd3129545152998 2023-04-03 05:56:50,018 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/49f0dc3625d3406b9df1e5a744fed6b9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/49f0dc3625d3406b9df1e5a744fed6b9 2023-04-03 05:56:50,019 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/056463fabd2546669641568e705e1e2e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/056463fabd2546669641568e705e1e2e 2023-04-03 05:56:50,022 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/a0a8f78c142146809182a67a9514b480, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/68bd3cf85f474b6aa02e1d22475b2f89, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/f10256f307874149bc27ef674ed18de3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1434b8975ce24ba8991e2e5aca1c5906, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/026a7f3aab984378951ad43674817a3a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ff5149cd32364b2d80acd45f8ae179f6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ae605e9fdde04b4dbdb0421768758184, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/a9614195fbb34f589b14c4bf80a0d585, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2e4f438169c2461b8326222fa800593c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/777386cc7e6345c1a70f28c369b6560b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/037ef8740b194b149590b56281b717f7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1fb2165687f347768337acf628462a92, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9d1f40326c2d4eefb71cfc160bed572c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8a07e24804954825a1a766d9806f58a9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2539dbf426a745acb84a9d132559bdee, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ee60214def5747759de3d4a433d69230, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b643e5f1bd9c4d5ea02de2ded030a780, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/3aac0100c0ff4d23a2376fdeafc8eb65, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/e858d4c0f59445e3beb2414c8c304775, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ff5c58d1539e455cbd5ec35e487249bd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9d24ef73b9e14e649de21cdc7db043f0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8dc3b4448c314871be536f68989d1f54, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/874422390c0648f5ad33ec35fc2fa0e6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/936656a29e68402a9c9f8498d510d430, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1af4ed1f1cdb4a3d979c1105026f5259, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6d6b1eef929a4d0d9c54919a58209899, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/010b2aa17d1b4468b246c0e3b0cf5bcd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/5ffce3f0d7d0473d8b4adf787e6ca699, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6921680f9dda4de2879cc425ef1c8496, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/01ce0b6db23c4d8aaf637e95dc59d83b, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b01ed9a1ce1d4039866996a6c38c779e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6d0db8e4013d4e2f93200060aa7e5126, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/542928d448344d0fb78bd1454ce9197a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9aa38ca72046417788fc6c1bf1d267ae, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6614470991b8460a8608b5a1c35469db, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/fac2313f975b49df83fbc54707fc4b4e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1f4935402fc94b939ff6afa2435712f6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2c4798afb9d5428da40cc8f7bdeba678, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8ce82c75e6074c1c85b063c82eb9d493] to archive 2023-04-03 05:56:50,023 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:56:50,024 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/a0a8f78c142146809182a67a9514b480 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/a0a8f78c142146809182a67a9514b480 2023-04-03 05:56:50,025 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/68bd3cf85f474b6aa02e1d22475b2f89 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/68bd3cf85f474b6aa02e1d22475b2f89 2023-04-03 05:56:50,026 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/f10256f307874149bc27ef674ed18de3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/f10256f307874149bc27ef674ed18de3 2023-04-03 05:56:50,027 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1434b8975ce24ba8991e2e5aca1c5906 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1434b8975ce24ba8991e2e5aca1c5906 2023-04-03 05:56:50,028 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/026a7f3aab984378951ad43674817a3a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/026a7f3aab984378951ad43674817a3a 2023-04-03 05:56:50,029 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ff5149cd32364b2d80acd45f8ae179f6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ff5149cd32364b2d80acd45f8ae179f6 2023-04-03 05:56:50,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ae605e9fdde04b4dbdb0421768758184 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ae605e9fdde04b4dbdb0421768758184 2023-04-03 05:56:50,030 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/a9614195fbb34f589b14c4bf80a0d585 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/a9614195fbb34f589b14c4bf80a0d585 2023-04-03 05:56:50,031 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2e4f438169c2461b8326222fa800593c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2e4f438169c2461b8326222fa800593c 2023-04-03 05:56:50,032 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/777386cc7e6345c1a70f28c369b6560b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/777386cc7e6345c1a70f28c369b6560b 2023-04-03 05:56:50,033 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/037ef8740b194b149590b56281b717f7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/037ef8740b194b149590b56281b717f7 2023-04-03 05:56:50,034 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1fb2165687f347768337acf628462a92 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1fb2165687f347768337acf628462a92 2023-04-03 05:56:50,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9d1f40326c2d4eefb71cfc160bed572c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9d1f40326c2d4eefb71cfc160bed572c 2023-04-03 05:56:50,035 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8a07e24804954825a1a766d9806f58a9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8a07e24804954825a1a766d9806f58a9 2023-04-03 05:56:50,036 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2539dbf426a745acb84a9d132559bdee to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2539dbf426a745acb84a9d132559bdee 2023-04-03 05:56:50,037 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ee60214def5747759de3d4a433d69230 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ee60214def5747759de3d4a433d69230 2023-04-03 05:56:50,038 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b643e5f1bd9c4d5ea02de2ded030a780 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b643e5f1bd9c4d5ea02de2ded030a780 2023-04-03 05:56:50,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/3aac0100c0ff4d23a2376fdeafc8eb65 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/3aac0100c0ff4d23a2376fdeafc8eb65 2023-04-03 05:56:50,039 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/e858d4c0f59445e3beb2414c8c304775 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/e858d4c0f59445e3beb2414c8c304775 2023-04-03 05:56:50,040 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ff5c58d1539e455cbd5ec35e487249bd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ff5c58d1539e455cbd5ec35e487249bd 2023-04-03 05:56:50,041 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9d24ef73b9e14e649de21cdc7db043f0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9d24ef73b9e14e649de21cdc7db043f0 2023-04-03 05:56:50,042 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8dc3b4448c314871be536f68989d1f54 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8dc3b4448c314871be536f68989d1f54 2023-04-03 05:56:50,043 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/874422390c0648f5ad33ec35fc2fa0e6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/874422390c0648f5ad33ec35fc2fa0e6 2023-04-03 05:56:50,044 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/936656a29e68402a9c9f8498d510d430 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/936656a29e68402a9c9f8498d510d430 2023-04-03 05:56:50,045 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1af4ed1f1cdb4a3d979c1105026f5259 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1af4ed1f1cdb4a3d979c1105026f5259 2023-04-03 05:56:50,046 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6d6b1eef929a4d0d9c54919a58209899 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6d6b1eef929a4d0d9c54919a58209899 2023-04-03 05:56:50,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/010b2aa17d1b4468b246c0e3b0cf5bcd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/010b2aa17d1b4468b246c0e3b0cf5bcd 2023-04-03 05:56:50,047 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/5ffce3f0d7d0473d8b4adf787e6ca699 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/5ffce3f0d7d0473d8b4adf787e6ca699 2023-04-03 05:56:50,048 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6921680f9dda4de2879cc425ef1c8496 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6921680f9dda4de2879cc425ef1c8496 2023-04-03 05:56:50,049 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/01ce0b6db23c4d8aaf637e95dc59d83b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/01ce0b6db23c4d8aaf637e95dc59d83b 2023-04-03 05:56:50,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b01ed9a1ce1d4039866996a6c38c779e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b01ed9a1ce1d4039866996a6c38c779e 2023-04-03 05:56:50,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6d0db8e4013d4e2f93200060aa7e5126 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6d0db8e4013d4e2f93200060aa7e5126 2023-04-03 05:56:50,052 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/542928d448344d0fb78bd1454ce9197a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/542928d448344d0fb78bd1454ce9197a 2023-04-03 05:56:50,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9aa38ca72046417788fc6c1bf1d267ae to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/9aa38ca72046417788fc6c1bf1d267ae 2023-04-03 05:56:50,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6614470991b8460a8608b5a1c35469db to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/6614470991b8460a8608b5a1c35469db 2023-04-03 05:56:50,054 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/fac2313f975b49df83fbc54707fc4b4e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/fac2313f975b49df83fbc54707fc4b4e 2023-04-03 05:56:50,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1f4935402fc94b939ff6afa2435712f6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/1f4935402fc94b939ff6afa2435712f6 2023-04-03 05:56:50,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2c4798afb9d5428da40cc8f7bdeba678 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/2c4798afb9d5428da40cc8f7bdeba678 2023-04-03 05:56:50,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8ce82c75e6074c1c85b063c82eb9d493 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/8ce82c75e6074c1c85b063c82eb9d493 2023-04-03 05:56:50,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e1c81ec703884187b9e36b87a64859f9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ee643df65a0046b1952dfd5d27a1d994, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b7099784cce1474ba600262025c938a8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/18f516b13e854a598fb07b46eb7c7a9a, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e91a0f935d7d49079bbd5a37a680aae2, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/cc934d3a535743c787ed56ae4aba47e0, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/07ed40064389470fac5bd5328b6072ad, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/1bf69a3444f3458eb96e2808c0130e7c, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/5d3975e77c0647f0adfd7c2f65fbc930, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e01de912468e4267929d78808739b733, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/407e05ccd0e045858c2253fa2b826d00, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/119a0ad1b9384cc99e1c5251944a1c81, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ac7f6381d2df42d8b113461c8b977958, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c31181aac6004c7a8242e794bfda60b1, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/177a09b04bb24efe830292c62fa2c289, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/6ef1c6a79667490a837af5998043a833, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c7c239a0a07b43468bef4e599d988f37, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/3fcd13cbcf72418fb894618a4e55c412, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/21878f72e5684efebe0e212ee618dc74, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7e59b21764a844ae84bdd5aee52124e7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ce5ca5ae619c475ba16538ea2af8b454, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/63309cb9e4704db3bbbc8df62c0a7ed8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7029e430805047fbb154036dd23bbfb6, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b506380221524b80bca3a0a2ba0b770e, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/0fab22a510f04530855196b53fadc8c8, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ecc113b745d04dde949be56e34450780, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/0df77ba5171e4bc58b972059d3ed1d77, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ca7207bc83d04712b6fe17a50abdaa34, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e92ba63c5067418b9bd2059a0b22a38f, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/272772f5653348a694cd1f3ff2496e52, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/290245ad647340ff8e2a25b30f1950cd, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/3f800a1f00984d1bb4fd9e386201e1b7, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/a07eddf9adff487c8a8e946610599275, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/32f43c42c5fa43ef932450e8b1693cc5, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/34ff5e02af3d47b1bc441c5fad1ba3d3, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c375aa13d8ac47b484d3a5bcad91ade9, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/a0147f052bd54e94a5e6558dc49f6765, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/98617d8b5baf43b5af95d72ebe87503d, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/1267d062e44640b0be99a94dc7229af5] to archive 2023-04-03 05:56:50,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-03 05:56:50,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e1c81ec703884187b9e36b87a64859f9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e1c81ec703884187b9e36b87a64859f9 2023-04-03 05:56:50,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ee643df65a0046b1952dfd5d27a1d994 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ee643df65a0046b1952dfd5d27a1d994 2023-04-03 05:56:50,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b7099784cce1474ba600262025c938a8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b7099784cce1474ba600262025c938a8 2023-04-03 05:56:50,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/18f516b13e854a598fb07b46eb7c7a9a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/18f516b13e854a598fb07b46eb7c7a9a 2023-04-03 05:56:50,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e91a0f935d7d49079bbd5a37a680aae2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e91a0f935d7d49079bbd5a37a680aae2 2023-04-03 05:56:50,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/cc934d3a535743c787ed56ae4aba47e0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/cc934d3a535743c787ed56ae4aba47e0 2023-04-03 05:56:50,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/07ed40064389470fac5bd5328b6072ad to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/07ed40064389470fac5bd5328b6072ad 2023-04-03 05:56:50,067 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/1bf69a3444f3458eb96e2808c0130e7c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/1bf69a3444f3458eb96e2808c0130e7c 2023-04-03 05:56:50,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/5d3975e77c0647f0adfd7c2f65fbc930 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/5d3975e77c0647f0adfd7c2f65fbc930 2023-04-03 05:56:50,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e01de912468e4267929d78808739b733 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e01de912468e4267929d78808739b733 2023-04-03 05:56:50,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/407e05ccd0e045858c2253fa2b826d00 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/407e05ccd0e045858c2253fa2b826d00 2023-04-03 05:56:50,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/119a0ad1b9384cc99e1c5251944a1c81 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/119a0ad1b9384cc99e1c5251944a1c81 2023-04-03 05:56:50,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ac7f6381d2df42d8b113461c8b977958 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ac7f6381d2df42d8b113461c8b977958 2023-04-03 05:56:50,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c31181aac6004c7a8242e794bfda60b1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c31181aac6004c7a8242e794bfda60b1 2023-04-03 05:56:50,075 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/177a09b04bb24efe830292c62fa2c289 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/177a09b04bb24efe830292c62fa2c289 2023-04-03 05:56:50,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/6ef1c6a79667490a837af5998043a833 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/6ef1c6a79667490a837af5998043a833 2023-04-03 05:56:50,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c7c239a0a07b43468bef4e599d988f37 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c7c239a0a07b43468bef4e599d988f37 2023-04-03 05:56:50,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/3fcd13cbcf72418fb894618a4e55c412 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/3fcd13cbcf72418fb894618a4e55c412 2023-04-03 05:56:50,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/21878f72e5684efebe0e212ee618dc74 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/21878f72e5684efebe0e212ee618dc74 2023-04-03 05:56:50,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7e59b21764a844ae84bdd5aee52124e7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7e59b21764a844ae84bdd5aee52124e7 2023-04-03 05:56:50,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ce5ca5ae619c475ba16538ea2af8b454 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ce5ca5ae619c475ba16538ea2af8b454 2023-04-03 05:56:50,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/63309cb9e4704db3bbbc8df62c0a7ed8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/63309cb9e4704db3bbbc8df62c0a7ed8 2023-04-03 05:56:50,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7029e430805047fbb154036dd23bbfb6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7029e430805047fbb154036dd23bbfb6 2023-04-03 05:56:50,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b506380221524b80bca3a0a2ba0b770e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b506380221524b80bca3a0a2ba0b770e 2023-04-03 05:56:50,084 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/0fab22a510f04530855196b53fadc8c8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/0fab22a510f04530855196b53fadc8c8 2023-04-03 05:56:50,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ecc113b745d04dde949be56e34450780 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ecc113b745d04dde949be56e34450780 2023-04-03 05:56:50,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/0df77ba5171e4bc58b972059d3ed1d77 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/0df77ba5171e4bc58b972059d3ed1d77 2023-04-03 05:56:50,086 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ca7207bc83d04712b6fe17a50abdaa34 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ca7207bc83d04712b6fe17a50abdaa34 2023-04-03 05:56:50,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e92ba63c5067418b9bd2059a0b22a38f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/e92ba63c5067418b9bd2059a0b22a38f 2023-04-03 05:56:50,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/272772f5653348a694cd1f3ff2496e52 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/272772f5653348a694cd1f3ff2496e52 2023-04-03 05:56:50,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/290245ad647340ff8e2a25b30f1950cd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/290245ad647340ff8e2a25b30f1950cd 2023-04-03 05:56:50,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/3f800a1f00984d1bb4fd9e386201e1b7 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/3f800a1f00984d1bb4fd9e386201e1b7 2023-04-03 05:56:50,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/a07eddf9adff487c8a8e946610599275 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/a07eddf9adff487c8a8e946610599275 2023-04-03 05:56:50,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/32f43c42c5fa43ef932450e8b1693cc5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/32f43c42c5fa43ef932450e8b1693cc5 2023-04-03 05:56:50,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/34ff5e02af3d47b1bc441c5fad1ba3d3 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/34ff5e02af3d47b1bc441c5fad1ba3d3 2023-04-03 05:56:50,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c375aa13d8ac47b484d3a5bcad91ade9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/c375aa13d8ac47b484d3a5bcad91ade9 2023-04-03 05:56:50,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/a0147f052bd54e94a5e6558dc49f6765 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/a0147f052bd54e94a5e6558dc49f6765 2023-04-03 05:56:50,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/98617d8b5baf43b5af95d72ebe87503d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/98617d8b5baf43b5af95d72ebe87503d 2023-04-03 05:56:50,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/1267d062e44640b0be99a94dc7229af5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/1267d062e44640b0be99a94dc7229af5 2023-04-03 05:56:50,097 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/recovered.edits/1013.seqid, newMaxSeqId=1013, maxSeqId=4 2023-04-03 05:56:50,098 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6. 2023-04-03 05:56:50,098 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 67f8cb6d380f875bf3bd14331600fad6: 2023-04-03 05:56:50,099 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:50,100 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=67f8cb6d380f875bf3bd14331600fad6, regionState=CLOSED 2023-04-03 05:56:50,100 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680501410100"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680501410100"}]},"ts":"1680501410100"} 2023-04-03 05:56:50,102 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-03 05:56:50,102 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 67f8cb6d380f875bf3bd14331600fad6, server=jenkins-hbase4.apache.org,37397,1680501214665 in 2.0330 sec 2023-04-03 05:56:50,103 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-03 05:56:50,103 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=67f8cb6d380f875bf3bd14331600fad6, UNASSIGN in 2.0360 sec 2023-04-03 05:56:50,103 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680501410103"}]},"ts":"1680501410103"} 2023-04-03 05:56:50,104 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-03 05:56:50,105 INFO [PEWorker-1] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-03 05:56:50,106 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0460 sec 2023-04-03 05:56:50,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-03 05:56:50,166 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-03 05:56:50,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-03 05:56:50,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:50,169 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:50,169 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:50,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-03 05:56:50,173 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:50,175 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C, FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/recovered.edits] 2023-04-03 05:56:50,177 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/424ded1ba3e043658f0377f753f3842e to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/424ded1ba3e043658f0377f753f3842e 2023-04-03 05:56:50,178 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/46de4850990b4e1d84b71c4d2d459d9a to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/46de4850990b4e1d84b71c4d2d459d9a 2023-04-03 05:56:50,178 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/47f3145c4f7c4761a9a12d4a9530f236 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/47f3145c4f7c4761a9a12d4a9530f236 2023-04-03 05:56:50,179 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/9722027742a24d19a5d2b28f3c74473f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/9722027742a24d19a5d2b28f3c74473f 2023-04-03 05:56:50,180 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/c805f326d74e48c4b34b2b24b7a89e09 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/c805f326d74e48c4b34b2b24b7a89e09 2023-04-03 05:56:50,181 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d68e2c1852b247a6a1c268669d084edc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/d68e2c1852b247a6a1c268669d084edc 2023-04-03 05:56:50,182 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/f6c5ee45d9d243858625fb867f7ed6cc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/A/f6c5ee45d9d243858625fb867f7ed6cc 2023-04-03 05:56:50,184 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/0772805e9ac14feeabb3418c11bb654f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/0772805e9ac14feeabb3418c11bb654f 2023-04-03 05:56:50,184 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/0ba69ac729334d008ba2995533552406 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/0ba69ac729334d008ba2995533552406 2023-04-03 05:56:50,185 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/3bbc71e5ac954df4bf292cae298ef2ca to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/3bbc71e5ac954df4bf292cae298ef2ca 2023-04-03 05:56:50,186 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/5ba1a7df2e05474ea43e5871ae03cebf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/5ba1a7df2e05474ea43e5871ae03cebf 2023-04-03 05:56:50,187 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b3d25e91d0f8465fbd51dec5603f2730 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/b3d25e91d0f8465fbd51dec5603f2730 2023-04-03 05:56:50,188 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/e2e86a01e059438ba719772bcc85e314 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/e2e86a01e059438ba719772bcc85e314 2023-04-03 05:56:50,188 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ecf022ac75b54a2ba46583d3fbc968fb to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/B/ecf022ac75b54a2ba46583d3fbc968fb 2023-04-03 05:56:50,190 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/05dc68d8ad6f45af8e1c166b9ea56212 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/05dc68d8ad6f45af8e1c166b9ea56212 2023-04-03 05:56:50,191 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7cb37ab84dce4b1aae8e965d837ddab5 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7cb37ab84dce4b1aae8e965d837ddab5 2023-04-03 05:56:50,192 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7e75c7c0a68544cc87d41680d3186558 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7e75c7c0a68544cc87d41680d3186558 2023-04-03 05:56:50,193 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7f23e030c3d94e6c8f536925235e18ac to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/7f23e030c3d94e6c8f536925235e18ac 2023-04-03 05:56:50,193 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/94893bf8809b4b3799de642f5090b2b9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/94893bf8809b4b3799de642f5090b2b9 2023-04-03 05:56:50,194 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b96b57a35bd04b9f8b770aee69225824 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/b96b57a35bd04b9f8b770aee69225824 2023-04-03 05:56:50,195 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ddfab1c72afd431f88b462af9ca45b24 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/C/ddfab1c72afd431f88b462af9ca45b24 2023-04-03 05:56:50,197 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/recovered.edits/1013.seqid to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6/recovered.edits/1013.seqid 2023-04-03 05:56:50,198 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/.tmp/data/default/TestAcidGuarantees/67f8cb6d380f875bf3bd14331600fad6 2023-04-03 05:56:50,198 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-03 05:56:50,198 DEBUG [PEWorker-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 05:56:50,199 DEBUG [PEWorker-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-03 05:56:50,201 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403017e3f952b394a99af95840581a35659 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403017e3f952b394a99af95840581a35659 2023-04-03 05:56:50,202 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403044ed791c42e4b67ab1c4371c1092615 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403044ed791c42e4b67ab1c4371c1092615 2023-04-03 05:56:50,203 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403164109c979784941981497255a868114 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403164109c979784941981497255a868114 2023-04-03 05:56:50,204 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031a8e0cc98346443cb89d4f22460a8354 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304031a8e0cc98346443cb89d4f22460a8354 2023-04-03 05:56:50,204 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040321959185de94484fbb57ba90a0fd164f to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040321959185de94484fbb57ba90a0fd164f 2023-04-03 05:56:50,205 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403260e6ed9f93448cd9a469045b2c087d0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403260e6ed9f93448cd9a469045b2c087d0 2023-04-03 05:56:50,206 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040326dad49441564d9588bc449a19af8915 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040326dad49441564d9588bc449a19af8915 2023-04-03 05:56:50,207 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040327555e5a5a404a4c84d639dcbefcc0c6 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040327555e5a5a404a4c84d639dcbefcc0c6 2023-04-03 05:56:50,208 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032c8d8b5a685f457c8c6e70448f74eccf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032c8d8b5a685f457c8c6e70448f74eccf 2023-04-03 05:56:50,209 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032f897130a385470e9878da4f7cad49a8 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304032f897130a385470e9878da4f7cad49a8 2023-04-03 05:56:50,210 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403320dc9e29e0d436bbaaf0211c7a741cf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403320dc9e29e0d436bbaaf0211c7a741cf 2023-04-03 05:56:50,211 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304033902931d8836425093df6be610b979f2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304033902931d8836425093df6be610b979f2 2023-04-03 05:56:50,211 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040342024907404844e597b3f6071187588c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040342024907404844e597b3f6071187588c 2023-04-03 05:56:50,212 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403447d2c30a6d645a6adf94ce3538894f2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403447d2c30a6d645a6adf94ce3538894f2 2023-04-03 05:56:50,213 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304035a9d7551cbb94a168529c4f549a13ae2 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304035a9d7551cbb94a168529c4f549a13ae2 2023-04-03 05:56:50,213 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304035efe2424cb954321b01e63ab884c07f9 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304035efe2424cb954321b01e63ab884c07f9 2023-04-03 05:56:50,214 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040361480bd1b9eb4c84a48d983f95a2a2cd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040361480bd1b9eb4c84a48d983f95a2a2cd 2023-04-03 05:56:50,215 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036509c97cfe354f21b941a0cbf8f47e1b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036509c97cfe354f21b941a0cbf8f47e1b 2023-04-03 05:56:50,216 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036c003761f67f41b89da0c021ab90f2d0 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304036c003761f67f41b89da0c021ab90f2d0 2023-04-03 05:56:50,217 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040377b0f887a89f4f6b9667787d7818ea64 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040377b0f887a89f4f6b9667787d7818ea64 2023-04-03 05:56:50,217 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037b95e323883e42258cc44d967389fa7b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037b95e323883e42258cc44d967389fa7b 2023-04-03 05:56:50,218 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037e14ce6955434bbcb9d1413f421ee724 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304037e14ce6955434bbcb9d1413f421ee724 2023-04-03 05:56:50,219 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040386ec2c3b422442fd9f9cff032a8f0d8d to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040386ec2c3b422442fd9f9cff032a8f0d8d 2023-04-03 05:56:50,220 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403937fc0f37f9848e2a03128f1f062383b to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403937fc0f37f9848e2a03128f1f062383b 2023-04-03 05:56:50,221 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b12de7e0e3e1461bb1bfcd037bf5f87c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b12de7e0e3e1461bb1bfcd037bf5f87c 2023-04-03 05:56:50,222 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b981475b2afb40f98a18fa23bd6369bd to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403b981475b2afb40f98a18fa23bd6369bd 2023-04-03 05:56:50,223 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403bd3346e23ac5444e81ab1985c3d4f210 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403bd3346e23ac5444e81ab1985c3d4f210 2023-04-03 05:56:50,223 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403bf9665895f874cfd8ae2a34691c91dfc to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403bf9665895f874cfd8ae2a34691c91dfc 2023-04-03 05:56:50,224 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c0abbf39fe5b4521b37c5e5b608e303c to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c0abbf39fe5b4521b37c5e5b608e303c 2023-04-03 05:56:50,225 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c24a26b5ce734537933690087d827e76 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403c24a26b5ce734537933690087d827e76 2023-04-03 05:56:50,226 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d0ddcb805463432b8826bf45d12d8b66 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d0ddcb805463432b8826bf45d12d8b66 2023-04-03 05:56:50,227 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d480de2cb203497e9df1b9943ef87682 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d480de2cb203497e9df1b9943ef87682 2023-04-03 05:56:50,228 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d6cd3c745d54451cbf363b81a761d737 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403d6cd3c745d54451cbf363b81a761d737 2023-04-03 05:56:50,229 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e4202b1c89b04ec5b86d0ce29543c1f4 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403e4202b1c89b04ec5b86d0ce29543c1f4 2023-04-03 05:56:50,230 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ee6ed93b83c4401f8df6ebfba56fadbf to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403ee6ed93b83c4401f8df6ebfba56fadbf 2023-04-03 05:56:50,230 DEBUG [PEWorker-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403fced55f9f46741339efa35fb20150ed1 to hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230403fced55f9f46741339efa35fb20150ed1 2023-04-03 05:56:50,231 DEBUG [PEWorker-5] backup.HFileArchiver(596): Deleted hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-03 05:56:50,233 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:50,234 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-03 05:56:50,235 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-03 05:56:50,236 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:50,236 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-03 05:56:50,236 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680501410236"}]},"ts":"9223372036854775807"} 2023-04-03 05:56:50,238 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-03 05:56:50,238 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 67f8cb6d380f875bf3bd14331600fad6, NAME => 'TestAcidGuarantees,,1680501376532.67f8cb6d380f875bf3bd14331600fad6.', STARTKEY => '', ENDKEY => ''}] 2023-04-03 05:56:50,238 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-03 05:56:50,238 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680501410238"}]},"ts":"9223372036854775807"} 2023-04-03 05:56:50,239 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-03 05:56:50,240 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-03 05:56:50,241 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 73 msec 2023-04-03 05:56:50,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=33301] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-03 05:56:50,270 INFO [Listener at localhost/37275] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-03 05:56:50,281 INFO [Listener at localhost/37275] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=226 (was 225) - Thread LEAK? -, OpenFileDescriptor=552 (was 544) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=372 (was 400), ProcessCount=169 (was 169), AvailableMemoryMB=3610 (was 3784) 2023-04-03 05:56:50,281 INFO [Listener at localhost/37275] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-03 05:56:50,281 INFO [Listener at localhost/37275] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-03 05:56:50,282 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45b2cf6e to 127.0.0.1:63744 2023-04-03 05:56:50,282 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:50,282 DEBUG [Listener at localhost/37275] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-03 05:56:50,282 DEBUG [Listener at localhost/37275] util.JVMClusterUtil(257): Found active master hash=1750681100, stopped=false 2023-04-03 05:56:50,282 INFO [Listener at localhost/37275] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,33301,1680501213465 2023-04-03 05:56:50,284 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-03 05:56:50,284 INFO [Listener at localhost/37275] procedure2.ProcedureExecutor(629): Stopping 2023-04-03 05:56:50,284 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-03 05:56:50,284 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 05:56:50,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-03 05:56:50,285 DEBUG [Listener at localhost/37275] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x219de320 to 127.0.0.1:63744 2023-04-03 05:56:50,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-03 05:56:50,285 DEBUG [Listener at localhost/37275] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:50,285 INFO [Listener at localhost/37275] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,37397,1680501214665' ***** 2023-04-03 05:56:50,285 INFO [Listener at localhost/37275] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-03 05:56:50,286 INFO [RS:0;jenkins-hbase4:37397] regionserver.HeapMemoryManager(220): Stopping 2023-04-03 05:56:50,286 INFO [RS:0;jenkins-hbase4:37397] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-03 05:56:50,286 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-03 05:56:50,286 INFO [RS:0;jenkins-hbase4:37397] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-03 05:56:50,286 INFO [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(3304): Received CLOSE for 6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:56:50,287 INFO [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:50,287 DEBUG [RS:0;jenkins-hbase4:37397] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0033bdb6 to 127.0.0.1:63744 2023-04-03 05:56:50,287 DEBUG [RS:0;jenkins-hbase4:37397] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:50,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 6f4c130726253b9eea5a0f54de82205c, disabling compactions & flushes 2023-04-03 05:56:50,287 INFO [RS:0;jenkins-hbase4:37397] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-03 05:56:50,287 INFO [RS:0;jenkins-hbase4:37397] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-03 05:56:50,287 INFO [RS:0;jenkins-hbase4:37397] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-03 05:56:50,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. 2023-04-03 05:56:50,287 INFO [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-03 05:56:50,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. 2023-04-03 05:56:50,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. after waiting 0 ms 2023-04-03 05:56:50,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. 2023-04-03 05:56:50,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 6f4c130726253b9eea5a0f54de82205c 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-03 05:56:50,288 INFO [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-03 05:56:50,288 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1479): Online Regions={6f4c130726253b9eea5a0f54de82205c=hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c., 1588230740=hbase:meta,,1.1588230740} 2023-04-03 05:56:50,288 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-03 05:56:50,288 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-03 05:56:50,288 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-03 05:56:50,288 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-03 05:56:50,288 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-03 05:56:50,288 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-03 05:56:50,290 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1505): Waiting on 1588230740, 6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:56:50,345 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-03 05:56:50,490 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1505): Waiting on 1588230740, 6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:56:50,690 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1505): Waiting on 1588230740, 6f4c130726253b9eea5a0f54de82205c 2023-04-03 05:56:50,699 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c/.tmp/info/9492f7a0cbd44e4e9e5cd8b2c0d48ade 2023-04-03 05:56:50,699 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/.tmp/info/23e106b89c6b4ad583b15c292e28d5da 2023-04-03 05:56:50,703 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 23e106b89c6b4ad583b15c292e28d5da 2023-04-03 05:56:50,703 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c/.tmp/info/9492f7a0cbd44e4e9e5cd8b2c0d48ade as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c/info/9492f7a0cbd44e4e9e5cd8b2c0d48ade 2023-04-03 05:56:50,706 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c/info/9492f7a0cbd44e4e9e5cd8b2c0d48ade, entries=2, sequenceid=6, filesize=4.8 K 2023-04-03 05:56:50,707 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 6f4c130726253b9eea5a0f54de82205c in 420ms, sequenceid=6, compaction requested=false 2023-04-03 05:56:50,710 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/namespace/6f4c130726253b9eea5a0f54de82205c/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-03 05:56:50,711 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. 2023-04-03 05:56:50,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 6f4c130726253b9eea5a0f54de82205c: 2023-04-03 05:56:50,711 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680501217295.6f4c130726253b9eea5a0f54de82205c. 2023-04-03 05:56:50,890 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-03 05:56:50,896 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-03 05:56:50,896 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-03 05:56:51,091 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-03 05:56:51,109 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/.tmp/rep_barrier/05d33268b1ba4099af16806a95a10f7f 2023-04-03 05:56:51,112 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 05d33268b1ba4099af16806a95a10f7f 2023-04-03 05:56:51,291 INFO [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1475): Waiting on 1 regions to close 2023-04-03 05:56:51,291 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740} 2023-04-03 05:56:51,291 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-03 05:56:51,491 DEBUG [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-03 05:56:51,518 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/.tmp/table/da6d69c35bce4120b8a5800d7b821b4a 2023-04-03 05:56:51,520 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for da6d69c35bce4120b8a5800d7b821b4a 2023-04-03 05:56:51,521 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/.tmp/info/23e106b89c6b4ad583b15c292e28d5da as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/info/23e106b89c6b4ad583b15c292e28d5da 2023-04-03 05:56:51,523 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 23e106b89c6b4ad583b15c292e28d5da 2023-04-03 05:56:51,524 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/info/23e106b89c6b4ad583b15c292e28d5da, entries=22, sequenceid=93, filesize=7.1 K 2023-04-03 05:56:51,525 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/.tmp/rep_barrier/05d33268b1ba4099af16806a95a10f7f as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/rep_barrier/05d33268b1ba4099af16806a95a10f7f 2023-04-03 05:56:51,527 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 05d33268b1ba4099af16806a95a10f7f 2023-04-03 05:56:51,528 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/rep_barrier/05d33268b1ba4099af16806a95a10f7f, entries=6, sequenceid=93, filesize=5.5 K 2023-04-03 05:56:51,528 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/.tmp/table/da6d69c35bce4120b8a5800d7b821b4a as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/table/da6d69c35bce4120b8a5800d7b821b4a 2023-04-03 05:56:51,530 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for da6d69c35bce4120b8a5800d7b821b4a 2023-04-03 05:56:51,530 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/table/da6d69c35bce4120b8a5800d7b821b4a, entries=9, sequenceid=93, filesize=5.5 K 2023-04-03 05:56:51,531 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 1243ms, sequenceid=93, compaction requested=false 2023-04-03 05:56:51,535 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-03 05:56:51,535 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-03 05:56:51,536 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-03 05:56:51,536 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-03 05:56:51,536 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-03 05:56:51,691 INFO [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,37397,1680501214665; all regions closed. 2023-04-03 05:56:51,699 DEBUG [RS:0;jenkins-hbase4:37397] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/oldWALs 2023-04-03 05:56:51,699 INFO [RS:0;jenkins-hbase4:37397] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37397%2C1680501214665.meta:.meta(num 1680501217034) 2023-04-03 05:56:51,703 DEBUG [RS:0;jenkins-hbase4:37397] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/oldWALs 2023-04-03 05:56:51,703 INFO [RS:0;jenkins-hbase4:37397] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C37397%2C1680501214665:(num 1680501216064) 2023-04-03 05:56:51,703 DEBUG [RS:0;jenkins-hbase4:37397] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:51,703 INFO [RS:0;jenkins-hbase4:37397] regionserver.LeaseManager(133): Closed leases 2023-04-03 05:56:51,703 INFO [RS:0;jenkins-hbase4:37397] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-03 05:56:51,703 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-03 05:56:51,705 INFO [RS:0;jenkins-hbase4:37397] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:37397 2023-04-03 05:56:51,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,37397,1680501214665 2023-04-03 05:56:51,709 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-03 05:56:51,710 ERROR [Listener at localhost/37275-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@6bcd7ffd rejected from java.util.concurrent.ThreadPoolExecutor@6ff2e65c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2654] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 05:56:51,710 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-03 05:56:51,710 ERROR [Listener at localhost/37275-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1ba2ee59 rejected from java.util.concurrent.ThreadPoolExecutor@6ff2e65c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2654] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 05:56:51,711 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,37397,1680501214665] 2023-04-03 05:56:51,712 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,37397,1680501214665; numProcessing=1 2023-04-03 05:56:51,713 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,37397,1680501214665 already deleted, retry=false 2023-04-03 05:56:51,713 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,37397,1680501214665 expired; onlineServers=0 2023-04-03 05:56:51,713 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,33301,1680501213465' ***** 2023-04-03 05:56:51,713 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-03 05:56:51,713 DEBUG [M:0;jenkins-hbase4:33301] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29644c37, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-03 05:56:51,713 INFO [M:0;jenkins-hbase4:33301] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,33301,1680501213465 2023-04-03 05:56:51,713 INFO [M:0;jenkins-hbase4:33301] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,33301,1680501213465; all regions closed. 2023-04-03 05:56:51,713 DEBUG [M:0;jenkins-hbase4:33301] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-03 05:56:51,713 DEBUG [M:0;jenkins-hbase4:33301] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-03 05:56:51,713 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-03 05:56:51,713 DEBUG [M:0;jenkins-hbase4:33301] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-03 05:56:51,713 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680501215755] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680501215755,5,FailOnTimeoutGroup] 2023-04-03 05:56:51,713 INFO [M:0;jenkins-hbase4:33301] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-03 05:56:51,713 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680501215754] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680501215754,5,FailOnTimeoutGroup] 2023-04-03 05:56:51,713 INFO [M:0;jenkins-hbase4:33301] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-03 05:56:51,714 INFO [M:0;jenkins-hbase4:33301] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-03 05:56:51,714 DEBUG [M:0;jenkins-hbase4:33301] master.HMaster(1512): Stopping service threads 2023-04-03 05:56:51,714 INFO [M:0;jenkins-hbase4:33301] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-03 05:56:51,714 ERROR [M:0;jenkins-hbase4:33301] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[IPC Client (1548992666) connection to localhost/127.0.0.1:37989 from jenkins,5,PEWorkerGroup] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-03 05:56:51,715 INFO [M:0;jenkins-hbase4:33301] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-03 05:56:51,715 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-03 05:56:51,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-03 05:56:51,715 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-03 05:56:51,715 DEBUG [M:0;jenkins-hbase4:33301] zookeeper.ZKUtil(398): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-03 05:56:51,716 WARN [M:0;jenkins-hbase4:33301] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-03 05:56:51,716 INFO [M:0;jenkins-hbase4:33301] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-03 05:56:51,716 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-03 05:56:51,716 INFO [M:0;jenkins-hbase4:33301] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-03 05:56:51,716 DEBUG [M:0;jenkins-hbase4:33301] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-03 05:56:51,716 INFO [M:0;jenkins-hbase4:33301] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 05:56:51,716 DEBUG [M:0;jenkins-hbase4:33301] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 05:56:51,716 DEBUG [M:0;jenkins-hbase4:33301] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-03 05:56:51,716 DEBUG [M:0;jenkins-hbase4:33301] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 05:56:51,716 INFO [M:0;jenkins-hbase4:33301] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.02 KB heapSize=393.72 KB 2023-04-03 05:56:51,812 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-03 05:56:51,812 INFO [RS:0;jenkins-hbase4:37397] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,37397,1680501214665; zookeeper connection closed. 2023-04-03 05:56:51,812 ERROR [Listener at localhost/37275-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@63a89d7f rejected from java.util.concurrent.ThreadPoolExecutor@6ff2e65c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2654] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 05:56:51,813 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): regionserver:37397-0x10148956f760001, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-03 05:56:51,813 ERROR [Listener at localhost/37275-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@468375ad rejected from java.util.concurrent.ThreadPoolExecutor@6ff2e65c[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2654] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 05:56:51,813 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@23954fcc] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@23954fcc 2023-04-03 05:56:51,814 INFO [Listener at localhost/37275] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-03 05:56:52,126 INFO [M:0;jenkins-hbase4:33301] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.02 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2701c6157c8e4f6eb9b348fc33b64a66 2023-04-03 05:56:52,128 INFO [M:0;jenkins-hbase4:33301] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2701c6157c8e4f6eb9b348fc33b64a66 2023-04-03 05:56:52,129 DEBUG [M:0;jenkins-hbase4:33301] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/2701c6157c8e4f6eb9b348fc33b64a66 as hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2701c6157c8e4f6eb9b348fc33b64a66 2023-04-03 05:56:52,131 INFO [M:0;jenkins-hbase4:33301] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 2701c6157c8e4f6eb9b348fc33b64a66 2023-04-03 05:56:52,132 INFO [M:0;jenkins-hbase4:33301] regionserver.HStore(1080): Added hdfs://localhost:37989/user/jenkins/test-data/e73ffd10-e9a5-3495-bbd3-895c3f2d1eaa/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/2701c6157c8e4f6eb9b348fc33b64a66, entries=65, sequenceid=578, filesize=24.7 K 2023-04-03 05:56:52,132 INFO [M:0;jenkins-hbase4:33301] regionserver.HRegion(2948): Finished flush of dataSize ~345.02 KB/353302, heapSize ~393.70 KB/403152, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 416ms, sequenceid=578, compaction requested=false 2023-04-03 05:56:52,133 INFO [M:0;jenkins-hbase4:33301] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-03 05:56:52,133 DEBUG [M:0;jenkins-hbase4:33301] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-03 05:56:52,136 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-03 05:56:52,136 INFO [M:0;jenkins-hbase4:33301] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-03 05:56:52,136 INFO [M:0;jenkins-hbase4:33301] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:33301 2023-04-03 05:56:52,138 DEBUG [M:0;jenkins-hbase4:33301] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,33301,1680501213465 already deleted, retry=false 2023-04-03 05:56:52,239 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-03 05:56:52,239 INFO [M:0;jenkins-hbase4:33301] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,33301,1680501213465; zookeeper connection closed. 2023-04-03 05:56:52,239 ERROR [Listener at localhost/37275-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@7ab70e0d rejected from java.util.concurrent.ThreadPoolExecutor@54f15a6d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4423] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 05:56:52,239 DEBUG [Listener at localhost/37275-EventThread] zookeeper.ZKWatcher(600): master:33301-0x10148956f760000, quorum=127.0.0.1:63744, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-03 05:56:52,239 ERROR [Listener at localhost/37275-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@538c6d89 rejected from java.util.concurrent.ThreadPoolExecutor@54f15a6d[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4423] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-03 05:56:52,241 WARN [Listener at localhost/37275] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-03 05:56:52,244 INFO [Listener at localhost/37275] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-03 05:56:52,351 WARN [BP-410271145-172.31.14.131-1680501210117 heartbeating to localhost/127.0.0.1:37989] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-03 05:56:52,351 WARN [BP-410271145-172.31.14.131-1680501210117 heartbeating to localhost/127.0.0.1:37989] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-410271145-172.31.14.131-1680501210117 (Datanode Uuid 483e758b-6075-4333-aa58-496ba5c29667) service to localhost/127.0.0.1:37989 2023-04-03 05:56:52,354 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/cluster_8445de51-c43b-f77a-d5f2-488a4722dd0f/dfs/data/data1/current/BP-410271145-172.31.14.131-1680501210117] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-03 05:56:52,354 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/ab43cf30-cb8b-2944-0902-3e1878be659b/cluster_8445de51-c43b-f77a-d5f2-488a4722dd0f/dfs/data/data2/current/BP-410271145-172.31.14.131-1680501210117] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-03 05:56:52,387 INFO [Listener at localhost/37275] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-03 05:56:52,559 INFO [Listener at localhost/37275] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-03 05:56:52,738 INFO [Listener at localhost/37275] hbase.HBaseTestingUtility(1293): Minicluster is down